miércoles, 17 de mayo de 2017

Descarregar programes de RAC1 desde la linea de comandes

Fa anys que sóc seguidor de "La Competència", i no sempre puc escoltarla en directe. Bé, hi ha la opcio de descarregar programes desde "rac1 a la carta".


El tema es que es un conyàs anar manualment a la web i descarregar el mp3. Fins aquesta setmana, el següent zsh script m'ha deixat descarregar el mp3 del dia:

function lacomp () {
    typeset -A days
    days[Sun]=Diumenge
    days[dom]=Diumenge
    days[Mon]=Dilluns
    days[lun]=Dilluns
    days[Tue]=Dimarts
    days[mar]=Dimarts
    days[Wed]=Dimecres
    days[mié]=Dimecres
    days[Thu]=Dijous
    days[jue]=Dijous
    days[Fri]=Divendres
    days[vie]=Divendres
    days[Sat]=Dissabte
    days[sab]=Dissabte
    #echo $(${days[$(date +%a)]})
    echo $(date +%a)
    wget "http://www.racalacarta.com/download.php?file=download.php-file=$(date +%m)$(date +%d)%2012h%20(${days[$(date +%a)]}%20$(date +%d-%m-%y))%20%20%20LA%20COMPETENCIA.mp3"
    return
    }

Doncs ara resulta que han canviat la web, i m'he hagut de fer un altre script per a descarregar l'últim programa de la competencia

function lacompeti () {
    curl http://www.rac1.cat/a-la-carta/la-competencia | pup 'a.fa-arrow-down attr{href} ' | head -1 | xargs wget
    ls -tr *mp3*download | tail -1 | xargs mplayer
}

jueves, 11 de mayo de 2017

Git 2.13 has been released

As you can see in github blog, there's a new git version (2.13).

There are some internal and some external changes, but the one that stroke me as most useful for me is conditional configurations.

Conditional includes, means that you can preset in your ~/.gitconfig some conditions, like

[includeIf "gitdir:~/work/"]
  path = .gitconfig-work
[includeIf "gitdir:~/play/"]
  path = .gitconfig-play
 
Now you can put whatever options you want into those files:

$ cat ~/.gitconfig-work
[user]
name = Serious Q. Programmer
email = serious.programmer@business.example.com

$ cat ~/.gitconfig-play
[user]
name = Random J. Hacker
email = rmsfan1979@example.com  

After that, you'll be able to commit away in any project in your computer, and git will do The Proper Thing.  This is just awesome, that all the hackery I had to simulate this can now go away.  I'm going to add this to my configs NOW. :)


As this is a git post, I'll take the opportunity to mention this blogpost about filter branch. Also, this one about packfiles. Both are advanced topics, but you know..... it's fun :)

jueves, 27 de abril de 2017

Pipeline arquitectures

Lately I've been developing a real time stream application, that delivers content via http long poll requests.

The details of the inner parts are not important, but overall, you can see it as many other applications, which transform, filter and distribute individual pieces of datastructures.  On the macro level, different components are connected via ZMQ or rabbitMQ, or any queuing system you like. Heck, you could use resque/sidekiq for that also.

Inside the components, although I like to use CLOS OOP (for it's inspection capabilities, and sometimes multiple dispatch), there's the idea of pipelines (or streams) which I like very much.

Here are some pointers for this approach I'd like to review:

-  Rob Pike talks. I love this one, which tells the difference about concurrency and parallelism. Also go concurrency patterns. I guess this one is also cool, but I haven't actually seen it, it just has a cool name.
- Talk about pipes in OCaml. This one isn't particularly enligthening if you have some knowledge of streams, as in SICP streams, but it gives a vision for functional programmers.
- Anything you find about pipeline arquitecture is ok.
- Thanks to native coroutines, lua provides nice uses and docs for pipelines. Also picolisp has support for coroutines.
- Full on functional. Coroutines for streaming in haskell.
- HighScalability article about disqus architecture.
- Also, using just active objects is a nice approach to, at least split responsabilities, and make the producer-consumer dance work. Kudos to the great CTM book that explains them super nicely.

martes, 25 de abril de 2017

debugging dns with dnsmasq

I'm not specially gifted at debugging network issues, but I try my best, and lately it seems I have to fight networks quite often.

This is how to debug dns queries using dnsmasq. Dnsmasqhas many many flags and options, but  with this I could log when requests are done, and their responses, so it gives you an understanding of what's happening under the hood.

  dnsmasq -k -q -h -R --server=8.8.8.8 --log-facility=-


Speaking about dns, here's a recent post by spotify, with more than you ever wanted to know about dns :)


sábado, 22 de abril de 2017

Tcl, Red, Nim, Spry, Wren and Gravity

On random walks around the internet, I'm always looking for a language that can provide me the easiness of use of shell languages (and pipes, don't forget pipes), but with more flexibility. I'm still using zsh (and trying to sneak picolisp and factor), but here are the new contenders/discoveries.

I found Red, which looked super interesting. Very smalltalkish, and with very good support for windows GUI programming and interacting with the outside world. I haven't done anything beyond the classical "hello worlds", "factorials", "guess number"... It looks very cool though. It probably has 0 libraries, but well, you can't have it all.  Refinements look very cool. no idea how they are implemented, but it's a really nice idea.

Tcl, which you usually read that is something like a 'bad joke', because everything is a string.  But since I read that antirez's article I think it has something to it.

Nim, is probably the language with more share of all the ones listed here. It's nice and cute, and compiles to native, or js (multiple backends always look cool). Although I'm learning C myself because it's one of the fundamental languages one has to know, nim looks like a useful alternative for more practical purposes. It supports lots of modern idoms (map, filter,.. reduce...) and practices (channels, etc..). And also has support for macros (shown in the link to convert a bf interpreter to a compiler).

Spry, is a language created by Goran Krampe, that smalltalk wizard. It's implemented in Nim, and has ideas from Lisp, Rebol, Forth and Smalltalk. It looks very nice, but it's just a toy for now.

Wren and gravity, both small, with ideas from Lua/smalltalk/erlang.

And in wren performance page, there's this paragraph:

"Tested against Lua 5.2.3, LuaJIT 2.0.2, Python 2.7.5, Python 3.3.4, ruby 2.0.0p247. LuaJIT is run with the JIT disabled (i.e. in bytecode interpreter mode) since I want to support platforms where JIT-compilation is disallowed. LuaJIT with the JIT enabled is much faster than all of the other languages benchmarked, including Wren, because Mike Pall is a robot from the future."


Mike Pall is a robot from the future.

jueves, 20 de abril de 2017

Logs, metrics and what to log.

Lately I've had to decide (and give some advice)  on logging and metrics.  I'm learning a lot as I go, but I think I'm starting to get some kind of grip of what should be what, and on what level.

Here's this link about logs vs metrics, which makes lots of sense. 

And also, a post about logging levels, and how a very basic rule serves for it deciding which level means what. Very useful info in the associated reddit thread.

  • DEBUG: information that is useful during development. Usually very chatty, and will not show in production.
  • INFO: information you will need to debug production issues.
  • WARN: someone in the team will have to investigate what happened, but it can wait until tomorrow.
  • ERROR: Oh-oh, call the fireman! This needs to be investigated now! 
Also, tools: lnav , oklog.

miércoles, 19 de abril de 2017

k/Apl info

  • APL self hosted compiler https://news.ycombinator.com/item?id=13797797
  • APL self hosted compiler screencast https://www.youtube.com/watch?v=gcUWTa16Jc0 
  • q manual http://code.kx.com/q4m3/
  • ok manual https://github.com/JohnEarnest/ok/blob/gh-pages/docs/Manual.md
  • k main page http://kparc.com/
  • kona https://github.com/kevinlawler/kona

domingo, 16 de abril de 2017

quote a day, some factor code and some bash+pup+jq

For todays hack, I wanted to have some tiny app that would display a quote/sentence/link at random. Basically, the same as the "fortune" unix command.

In principle it should be trivial, it's just picking a random quote from a huge file.

The idea of doing it in factor was, well... to be able to do anything at all in factor, as most of my previous attempts failed to produce anything that even did what I expected. The tiny repo is in github.

What made Xerox PARC so special?

Via Hacker News, I found this cool answer by Alan Kay about "What made Xerox PARC so special?"

Very nice, and I got a couple of book references I'm already pushing in my to-read.

EDIT: I just read that Bob Taylor has died today at 85. :(

sábado, 1 de abril de 2017

TIL: Amazon Nat Gateway drops your connections after 5 mins idle

Yes, after many hours of head scratching, I found out that if you're inside an amazon VPC, and use a NAT gateway or a NAT instance as your internet gateway, silent connections for more than 5 minutes will be dropped, or, even "better", just silenced, but you won't notice anything, just silence.

For long streaming connections going 'mute' for maintenance, you won't recover automatically.

miércoles, 15 de marzo de 2017

Browsing allegro lisp docs inside emacs

While programming, I love to read documentation using helm-dash. I've created quite a few docsets for apps/languages that lacked them in the official repos.

Currently I was using python, common lisp, Picolisp, and Redis, but I felt I was going too often to the franz's Allegro Lisp documentation site.  It's nice and loads fast, but it doesn't provide any useful search.

So I created a dash docset for Franz's Allegro Lisp.  As usual, it's in my github.

And here's a little gif I recorded to see how does it look.


martes, 14 de marzo de 2017

git reflog with dates

reflog is this superhero that only appears when it has to save your ass.

A neat thing about it is that it logs everything you do in a given machine. It works locally, none of the reflog is pushed anywhere, and if you 'git clone', your reflog is empty. But...

For us, that are work are doing very far from BlueGreenDeployments, by doing git pull in a server and reloading (in Common Lisp planet you do hot reloads like a boss), this gives a track of when did pulls happen, and so you can find re-imagine the history of deploys, pulls, deploys, fixes.

It's a suboptimal solution, but given it's for free "don't look a gift horse in the mouth".

As usual in the class of git commands that list info, there are lots of formats and customizations.  For me, a simple "git reflog --pretty=oneline --date=short" does the job. 

domingo, 12 de marzo de 2017

CTMCP finished!

After a big hiatus of 6 months (space for CLtL The  unix programming environment, Object-Oriented Programming in Common LISP
,To mock a mockingbird (part of it) , I continued reading Concepts, Techniques, and Models of Computer Programming.

And it's so good I'm gonna put it next to SICP, GEB, HOP and PAIP (when I get around finishing it).  Here are some highlights:

  • The book covers a huge spectrum of programming techniques, and ponders lots of variants and different ways of achieving different tasks.  It focuses a lot in declarative programming, which comes very natural for extending the model to streams, logic, relational and constraint programming.  
  • The whole book examples are written in Oz/Mozart, which is a multi-paradigm language with a Prolog-ish syntax which seemed quite odd in the beginning but in the end I learned to like.
  • The tone of the book is so unopinionated and so 'formal', that I missed Eva Lu Ator, and Alyssa P. Hacker. 
  • I had a lovely AHA moment when discovering difference lists. Very useful to understand the paradigm shift of Prolog style programming.
  • Concurrency is touched quite heavily, and the explanations on why the different models exist, and when to use what were very useful from a practical perspective.  I loved how they introduced Active Objects.
  • Explanations of 'well known' concepts like recusion, accumulators, mapcar, transducers, closures, and streams are very nice, and made me remember of the times I discovered them for the very first time.  
  • Overall, the book is very readable. Clear style and vocab makes it readable in bed.
  • There's an introduction to Constraint Programming. Which feels like total magic. I guess it deserves reading a whole book on that to know a bit more about it.
Really, if you tried SICP and/or HOP and liked it, this book will provide 940 pages of fun and enlightenment.

Next, I'm reading PoEAA (a bit old and enterprisey but it's Fowler). And I have Beautiful Data (O'Reilly) waiting....




viernes, 10 de febrero de 2017

Zsh global aliases for command chaining

(The motivation for this post is my recent discovery of oilshell, a very promising new shell.  The author has been posting regularly for some time, and he has quite insightful ideas. Some of them I had them too and I didn't know they conformed to some defined concepts (I discovered Bernstein chaining two weeks ago). Whatever.... here it goes, as a way to open more discussion topics)

There's a small zsh trick that is unmatched by any other shell. It's maybe just syntax sugar, but it enables some pretty neat tricks.

The feature is global aliases (or "alias -g").

They are alias substituted anywhere on a line. Global aliases can be used to abbreviate frequently-typed usernames, hostnames, etc.

I think of them as some kind of reader macro, that applies substitutions to strings at read time, so there's no need to substitute a full command by another full command.

alias -g

 

The simplest case is to make an alias for a bunch of flags for a command you use many times (Also look at zsh globbing to see other examples of 'simple' alias -g).

alias -g GLWEEK=' --since=1.week.ago --author=kidd'

Given this ("git last week"), we can use it in different contexts. In this example, the only useful ones that come to my mind are "git log" and "gitk", but to get the feeling of what it does internally, you can 'echo GLWEEK' and you'll get the flags printed.  It allows for a lot of composability, and Bernstein chaining because everything concatenates and tries to apply.

Piping through

 

Another way to see it is as a way to Forthify your shell a bit more with pipes. This is how I use the feature the most:

If you make aliases expand to pipes (left, right or both), you can make many unix commands compose in a seamless way. Under the hood, you know that the connexion happens through stdin/stdout, but visually it eliminates most of the clutter.

Let's see some examples:

alias -g T='| tail'
alias -g T1='T -1'
alias -g X='| xclip '
alias -g G='| grep '
alias -g GV='| grep -v '
alias -g DM='| dmenu '

Let's see a few ways we can compose this:

ls -ltr G foo GV bar T1 X
evince $(ls *pdf DM)

With some training, it's very easy to see that this will copy the last file that matches 'foo' but doesn't match 'bar', or it will open the selected pdf file.

XARGS

 

There are some commands that do not accept arguments from stdin, but they require them to be as command line arguments. in those cases, one starts doing things like:

mplayer $(ls *(mpg|mp4|avi) DM )

Which works, but it breaks the pipe flow.  For that, we can use xargs (as we saw a few posts ago), and we can use, again, another global alias:

alias -g XA='| xargs '

now we can do "ls *(mpg|mp4|avi) DM XA mplayer" .

FUNCTIONS 

 

Here's an example of a uniq function on steroids. it accepts a field number to uniquify, and the rows do not have to be sorted to work.

function uc () {
    awk -F" " "!_[\$$1]++"
}

alias -g UC=' | uc '
alias -g P1='| awk "{print \$1}"'

Let's see how to use this one: For the sake of the example, let's pretend we want to know the users that have some process running in my machine.

ps -axuf GV USER UC 1 P1

Of course, this is the same as typing the vanilla commands yourself, but IMO, it makes quite a lot of difference on the experimentation easiness.

There are some more fancy tricks, but we'll leave them for some future post.

The whole point of this is that some features allow for greater composability, and let you build your ad-hoc tools in a much easier way than others.  Oilshell has plans to have '{', '}' and '[', ']' as operators, so you can have block like syntax (maybe the semantics of tcl's quoting would be enough?).  Anyway, it's great that the shell space is active and there are interesting things happening. I'll keep oil in my radar :)




miércoles, 8 de febrero de 2017

Henry G Baker's papers

I was rereading "Out of the tarpit" paper, and by the end of it I noticed they reference a paper from Henry Baker: "Equal rights for functional objects or, the more things change, the more they are the same.". The author's name rang a bell, and I decided to explore a bit more.

So, He's the author of a paper I loved to read 3 months ago or so: Pragmatic Parsing in Common Lisp.

The article is an adaptation of Val Schorre's META-II paper to common lisp, leveraging reader macros to create parsers in very few lines of code.

Reading a bit more, I found this other very enjoyable article: Metacircular Semantics for Common Lisp Special Forms . This one takes the approach of metacircular interpreters to explain the behaviour of some common lisp special forms in terms of other common lisp expressions.  I love the bottom up approach, and reminds me of "git from the bottom up", or CTMCP's explanations of the extensions to the Oz language.

He wrote lots of articles related to Lisp to different degrees, but most of them (the ones I skimmed) are written in a very clear language (no pun intended), and quite fun and enlightening reads.

domingo, 22 de enero de 2017

Autocomplete erc nicks with @ sign (for slack)

 This is a very quick hack for something I've wanted for some time.  I'm using slack through their IRC gateway. That allows me to use erc, yasnippets, bug-reference-mode,  and whatnot.

The few things that annoy me of that interface is that to ping users, in slack you are expected to prepend the nick with an at sign. That makes autocompletion kinda useless because, as this redditor explains , it's a pain in the ass to type the nick, autocomplete, go back, add the @ sign, and move forward.

I haven't found a way to solve this in an elegant way, but as a hack, I could add tihs advice functions that will understand @nick as well as 'nick' when autocompleting.  I'd love to be able to just type the name and the autocompletion framework fill the '@'  sign itself, but I couldn't find it, so I'll try this approach I got so far and see if it works well enough for me.

I hope it's useful for some of you, dear emacslackers :)

sábado, 21 de enero de 2017

using xargs to parallelize command line

Did you know that xargs can spawn multiple processes to execute the commands it gets? It's the poor man's gnu parallel (very poor).

Well, I've been dabbling quite a bit with shell scripts lately and I've come to this balance of using shellscripts for simple (or not so simple) tasks. It's a mix of suckless and Taco Bell programming....

Well, the problem at point was to create many connections to some api endpoints. The endpoints would be serving data in a streamming fashion (think twitter hose).   My idea would be to spawn several curls to a "circular" list of urls.

I thought about wrk (because lua), vegeta or siege, but I'm not sure how they would cope with 'persistent' connections, so I tried my luck with plain curl and bash.

It's funny how you can solve some issues in bash with so few lines of code.  I don't have stats, or anything, but the only thing I wanted, that was to generate traffic can be done in plain simple shellscripting.

Things I learned:

- ${variable:-"default-value"}   is great
- doing arithmetic in bash is next to impossible.
- xargs can spawn multiple processes, and you can control the number, so you can use the number as an upper limit.
- while true + timeout are a very nice combo when you want to generate any kind of 'ticking' or 'movement' on the loop.

Here's the script:

jueves, 19 de enero de 2017

git contribution spans (emacs case study)

 Let's play a little, just for the fun of it, and to gather some metainformation about a codebase.

I sometimes would like to find out the commit spans of the different authors. I may not care about the number of commits, but only first and last. This info may be useful to know if a repo has most long time commiters or the majority of contributors are one-off, or one-feature-and-forget.

First we'll sharpen a bit our unix chainsaw:

Here's a little helper that has proven to be very useful. It's similar to uniq, but you don't need to sort first, and also it accepts a parameter meaning the column you want to be unique. (Unique by Column)

function uc () {

    awk -F" " "!_[\$$1]++"

}


And then, you only have to look at man git-log to find out you can reverse the order of the logs, so you can track the first and the last appearence of each commiter.


git log --format='%aE %ai'  | uc 1 | sort  >/tmp/last

git log --reverse --format='%aE %ai -> '  | uc 1 | sort >/tmp/first

paste /tmp/first /tmp/last > /tmp/spans.txt


Here is the output file. it has some spurious parsing errors, but I think it still shows how powerful insights we can get with just a bit of bash, man and pipelines.

Just by chance, the day after I tried this, I discovered git-quick-stats, which is a utility tool to get simple stats out of a git repo. It's great that I could add the same functionality there too via a Pull Request :).

Big thanks to every one of the commiters in emacs, being long or short span and amount of code contributed. Thanks to y'all 

gnu parallel as a queuing system

This post is about a gnu parallel, a tool I recently discovered, and I'm starting to use a bit more every day.

At its core, it's a command to multiple commands in parallel, but it has many many different options to customize how the paralelization is done, notifications, and other configs. Take a look at the official tutorial or the man page, which contain a wealth of info and examples.

Let's get SICP videos

The use case I have today is to use it as a simple queuing system. I just want processes to start when I have a new job for them.  The task at hand is to download all sicp lectures, at one download at a time (don't want to hog the network).

First of all, we notice the pattern of the links:

  1. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec1a.mp4
  2. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec1b.mp4
  3. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec2a.mp4 
  4. http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec2b.mp4 

We notice the pattern, right? Let's craft some generator for the filenames.


 perl -e 'for(1..15){for $i (('a','b')){print "wget http://www.archive.org/download/MIT_Structure_of_Computer_Programs_1986/lec$_$i.mp4\n"}}' >sicp.list


After we generated the file, we're going to run the command in the following way:

cat sicp.lisp | parallel -j1 --tmux

this makes parallel run one after the other, and just putting the outputs of each job in a tmux tab.

B, b, but.... what's the point of all this?

Ok, we didn't use parallel for anything useful, we could have run the list as a shellscript, and be happy.   The idea is that we can use this simple mechanism to treat the file as a job queue, that waits for new incomming jobs and then processes them.  Beware, it has very little logging, and you can't do very sophisticated error recovery (see --resume-failed), so it's NOT a replacement for resque/sidekiq/etc...  In fact, I'd love to see something like a suckless queuing system based on parallel and bash.

So, here's the command to use this tool as a queuing system.

touch joblist; tail -f joblist | parallel --tmux 

Then, add lines to the joblist for them to be executed. It's a very easy plumbing task:

echo "sleep 10" >joblist

tail -f works as our event loop, waiting for new tasks to come, and it will pass the tasks to parallel, that will apply the job contention depending on the number of jobs you configure it to run in parallel.

I've just scratched the surface of what parallel is able to do. Do some searching around, and take a look at the man and tutorial to get a grasp of what this amazing gnu tool is able to do.  Taco Bell programming at its best!

References

Here I'm pasting some useful refs (appart from the ones already mentioned).