GoFCCYourself — Setting up your workbench (elasticsearch/kibana)

Great, we’ve got some data thanks to the previous post. Now how do we dive in?

Well seems like these filings were originally in a document database, and given their json content, let’s pick on at random from the wonderful list from wikipedia.

I know the names of some of these, and surprised other one’s weren’t on this list (Cassandra and CouchDB, guess apache projects got off the short-list somehow…).

Anywho, I’ve heard good things about elasticsearch and a duckduckgo search leads me to believe it should do at least some of what I want, there seem to be reasonable docs, and Kibana looks cool. I’m sold (mostly cause it’s free/open)! Haven’t touched java in years let’s hope I don’t need to get in the weeds at all.

GoFCCYourself — stange data, let’s capture it!

So this is interesting… yadda-yadda-yadda net-neutrality is gonna be taken away again. That is completely silly (imho) for various reasons:

• I don’t want a fragmented/silo’d web
• Information is the most valuable resource — literally what the internet is — putting a tollway on it only increases the gap for the less advantaged

So I was never really interested in the politics of it all. People have varying viewpoints, or do they? Let’s dive into the data.

3 quick tips for lxd

Turn on completion in zsh

Include the following lines in your .zshrc

autoload bashcompinit
bashcompinit
export -f _have() { which $@ >/dev/null }
source /usr/share/bash-completion/completions/lxc

This turns on ability to use bash completions within zsh.

Then we need to create our own function for the _have bashism that is typically defined in /etc/bash_completion.

Finally just source the already awesome tab-completion for zsh.

launch tmux in a lxc

A slightly more involved version of this.

function lmux() {
    lxc exec $1 -- sh -ic 'WHO=$(awk  -F":" "\$3>999&&\$1!=\"nobody\" {print \$1; exit 1}" /etc/passwd || getent passwd 0 | sed "s_:.*__"); su -c "cd ~; script -qfc \"tmux attach\" /dev/null" $WHO'
}
function lmux-autocomplete {
    reply=( $(lxc list -c "n" | grep -v '^+' | tr -d '| ' | tail -n +2) )
}
compctl -K lmux-autocomplete lmux

It gets the first user from /etc/passwd or falls back to root, changes to the home directory of that user and runs “tmux attach” via script which creates a tty for us.

Since we obviously want autocomplete, write a small function to do that and enable it.

Use dns names for your lxcs

If you are using NetworkManager on your machine where lxd is running and want dns names for your containers. Simply run the following:

echo server=/lxd/$(lxc network get lxdbr0 ipv4.address | sed 's/\/.*//') | sudo tee /etc/NetworkManager/dnsmasq.d/lxd.conf
sudo service network-manager restart

This assumes that you are using managed networking so that there is a line like server=/lxd/10.232.128.1 (the dnsmasq config line to add a dns server authoritative for the “lxd” TLD from that ip — where the lxd daemon has its own dnsmasq for the managed network).

So now if you can reach your lxcs like so:

$ lxc launch ubuntu cooltest
$ ping cooltest.lxd
PING cooltest.lxd (10.232.128.58) 56(84) bytes of data.
64 bytes from 10.232.128.58 (10.232.128.58): icmp_seq=1 ttl=64 time=0.046 ms
64 bytes from 10.232.128.58 (10.232.128.58): icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 10.232.128.58 (10.232.128.58): icmp_seq=3 ttl=64 time=0.043 ms

And use that in your browser if you are doing something. Never have to remember ips again!

#! with multiple args

Today I ran into the fact that shebangs with multiple arguments do not get parsed as I would expect…

In order to prevent multiple copies of a long-running cronjob from running simultaneously I wanted to use flock. So I changed the first line of my script.

#!/bin/bash

My standard above, to something that I would expect to work.

#!/usr/bin/flock --nonblock /tmp/mylockfile /bin/bash

Running it gave me:

/usr/bin/flock: unrecognized option '--nonblock /tmp/mylockfile /bin/bash'

Followed by flock’s usage information…

Not exactly what I was hoping for. After a bit of duck-ducking I found some helpful information[1,2] which taught me the following:

• The shebang is used by the kernel to call the correct interpreter if the leading magic bits of the file are “!#”
• Most historic kernels used to only pass the first 32 bytes of the line, Linux uses 127. So watch that character count!
• Only the first space is used to separate the command from the argument, all trailing spaces are ignored and passed as a single argument, essentially an exec('prog', 'arg1 arg2 arg3')
• Perl parses its arguments as it pleases, and doesn’t let the exec string tell it what to do.

I ended up with the following, which while it works was a bit too ugly for me to commit for others to review:

#!/usr/bin/perl -e exec "/usr/bin/flock", "--nonblock", "/tmp/mylockfile", "/bin/bash", @ARGV

I also have a simple program as a perl gist, simply because I haven’t touched perl in years, so if I need to in the future my shebangs can be a bit simpler.

#!/bin/spacexec /usr/bin/flock --nonblock /tmp/mylockfile /bin/bash