Blogging On The Go With Jekyll, Jenkins, Git, And A Mobile.

Jekyll, Continuous Delivery, And A Connected Smartphone.

For most of February, my daily posts have been lacking here. I’ve been writing them though. But, publishing them here has had to wait. Because, I was in the process of changing the entire workflow of how this blog operates/updates; in the process of changing everything about the workflow that I had created last year, because it was very limiting when I was out, and traveling; I do love to travel.

Yep, I was growing tired of not being able to publish posts on the go. Sure, I could write them (and I write them daily) but there were times when I wished I could just publish them off of my phone or for that matter - write, and publish them irrespective of the kind of connected device (device independent) I could use. I am glad that now I can.

Limitations Of Jekyll, For Me.

I love the idea of a static site generator. Last year, moving from Tumblr to a self hosted, statically generated blog was definitely one of my good calls. It’s been super fun, playing around, and getting to know Jekyll. And, so far I’ve not had much of a complaint, except for the speed. The other things, w.r.t Jekyll, that I felt were limiting me, were the fact that I could not publish the posts; those which were written while traveling, and the manual tasks of building, deploying, and committing the blog on a private github repository, post publication. I had known right from the onset that the manual tasks could be automated. While I could write up scripts or cronjob, on my home desktop, those that could take care of automating the process, these would be limiting my publishing again, when traveling. Not to mention that if I did venture that way, it would make me more dependent on a single machine to get these posts out.

Sidenote:

I did give some thought to moving over to WordPress, before I actually sat down to get all of this done. But, I do like to experiment. And, WordPress somehow doesn’t still feel right for me. Sure, it has a mobile app from which you can write, and publish directly to your WordPress blog. But, it’s not something that I want to use. Same goes for Ghost, etc.

Enter: A More Central Workflow (Think Cloud)

This is when I decided that I wanted to move the entire workflow of publishing using Jekyll to a server, and have it build, deploy (publish) and commit without much of interaction from my end, apart from just writing my posts. I also knew that figuring this workflow out would take some time.

Additions to the centralised workflow

While I was figuring things out, I was also using some awesome tools to test the usability of the old design, and the experience it offered you — the reader.

The result of all that is this new design, new logo, and a new brand name/identity.

Here’s How This Blog Works.

After playing around with quite a few tools, services, and technologies in general, I decided to go ahead with having the entire Jekyll setup on my server, using Jenkins, and a few of its plugins to build, and deploy the blog regularly.

Along with Jekyll, and Jenkins, I also use a few iOS apps (mentioned below) to speed up the process of putting the post(s) in the \_posts folder.

In essence to publish a post, here on this blog, all I’ve to do is write it, and include some images. I can write a post from anywhere, on any connected device; sometimes, even write, and access the post directly from/on the server.

Shifting Everything To The Cloud:

‘Cloud’ is a fancy term, I’d like to think so. It’s just a synonym for a different type of a ‘Shared Server’ (one where resources are shared using virtualization) that’s accessible from everywhere, given that you’ve access to it. This server, essentially is a file server that contains all the files, and then various apps can sync, and connect with it depending on the way it’s setup. That’s the definition of cloud for me; para-virtualisation, hvm, etc are all relative so to speak.

Cloudy Options:

When I was considering a central repo of sorts for the raw markdown files, I had mind-mapped some options that would be ideal to host these posts in their nascent format. These were:

1. Mega.nz: My first consideration was Mega. Free 50GB of storage space, did sound awesome. I would’ve gone ahead with it, but as I mention below I wanted to save bandwidth, time, and other overheads while fetching the markdown files. This point is true for the other options below too, except for the webserver where the blog resides.
2. Google Drive: 15GB of free space. I did not want to open yet another gmail account, exclusively for just syncing my files.
3. Dropbox: 2GB of free space. Now, Dropbox was the closest thing to hosting all those files on the webserver. I would’ve opted for Dropbox, but the collective size of the raw files (posts in markdown + images, videos) required for generating the static blog posts in Jekyll, are well over the 2GB limit. I could’ve used Dropbox sync, and synced all files directly to the build folder on the server as they changed, or got created. This is true for Mega too, but I decided against doing that.
4. Box.net: 50GB of enterprise level storage, free. Same concerns listed above for Dropbox, and Mega.
5. A private Git repository or GitHub Pages: I deferred from using a private repo as the ‘master’ node; I did have to issue a pull request before every build, and then a commit after every successful one. Now, this would’ve totally made sense to me if there were more people involved with respect to writing, and publishing posts on my blog. But, pulling from the repo while there’s just me, didn’t make sense. I considered GitHub Pages too, to host, and sync everything; but I do like writing, and using my own custom Jekyll plugins. So, GitHub Pages was out.
6. This webserver: Hosting all files, directly on the web server made sense. The choice was made in order to reduce any external API calls, etc. to other cloud based services mentioned above; thus reducing overheads, and additional external requests before the build process even started. Now, there are downsides of going with this option too. For e.g deciding where to write the post - directly onto the server using WebDAV, SFTP, etc or on the local / client node; uploading these raw markdown, and associated media files using SFTP was a pain. It’s a pain. But it’s an advantage too, if you change the perspective. Another disadvantage is that as this blog grows, it will take up more space. It will take more CPU/RAM/resources to build these static pages. This would force me to upgrade the server as time goes on, and as this blog becomes bigger. Which means, that I would’ve to spend more every month. And for now, I am okay with that.
7. NAS: A lot of concerns here. I could use a Rpi that I already own to deploy a NAS server, and then use it as a master node for syncing, building, and deploying the build artifact. But that would require the Rpi to be connected to the Internet 24/7, for me to be able to access it. I do live in India; the Internet has gotten better here, but it is still subject to a lot of disconnects. Security is another concern to think about. Also, to build, and then deploy this NAS would’ve taken me a long time which I do not have the luxury of. I am keeping this option open, and maybe in the future I may very well go ahead, and deploy one.

Jenkins For Continuous Integration & Delivery.

Once I decided to store all the files, and packages required to generate this blog with Jekyll on the server, I needed a Continuous Integration, and Continuous Delivery tool(s) that could run the build process, update the folder where the blog resides with new content, and then commit the changes over to a private git repo for keeping an ever fresh backup.

CI & CD Options:

1. Integrity (Integrity App): This was my first choice. Since integrity is written in Ruby, it would mean that I would not have to install additional libs/compilers, etc. It would also mean that I would have the freedom of tailoring it the way I wanted to. But, because the build, and delivery system needed to be on the web server directly, I had to give this up. Installing Phusion Passenger, securing it, would be additional tasks. And writing the build script, would take time here too, because of the slight but very visible learning curve involved. So, I had to give this up in favour of Jenkins.
2. Travis CI: Travis is a SaaS app offering for CI. Two versions exist, one at travis-ci.com, the other at travis-ci.org. The .com is for private repos, and the .org is from the opensource ones. I was considering Travis CI, earlier on when I wanted to host, and build the entire blog from a git repo - a commit would trigger a build, in this case. But, I opted out because the workflow changed. I wanted to write, and be connected to the server at the same time. Travis CI, also involved a learning curve for me, since I’ve had no experience with it.
3. Jenkins: I’ve had some previous experience with Jenkins/Hudson, so it made sense to use Jenkins, hosted on to the server directly. It’s relatively easy to setup, and can be served via Apache in a simple manner. Jenkins, is mostly a trigger mechanism for builds. ANT, and Maven are the build, and delivery tools that are used, to actually create artifacts a.k.a build results/compiled source code/software, and deploy them. And for a simple task comprising of a Jekyll command, rsync-ing of the _site folder with the publicly readable blog folder, and committing the successfully built artifact(s) to a private git repo for backups - Jenkins provides the option to build via shell script(s). That’s the way I went. I wrote a small bash script that does all the building, syncing, and committing on a successful build. Jenkins is pretty extensible, and has great community of developers behind it. This has led to many Jenkins plugins being available for use. I use the Files Found Trigger plugin, which keeps an eye on a file(s) or a folder for any kind of modification. You can read about FFT Trigger here.

The bash script I use (a version of it) is embedded below. Do note however, that the script may be different for you, if you do choose to go ahead with Jenkins, and a Jekyll setup.

There are other, free, and open source options. If you’re interested in looking up those, you can have a look at the list of all the CI tools available here, and give this post a read too.

Mobile Apps Used To Publish Blog Posts.

Now that the server part has had been setup, and done. The builds were coming along fine, everything’s working the way it’s supposed to on the server, I needed to be able to write/edit a post, create, and edit media content that gets used in every post on this blog - like the header images, then be able connect to the server, and upload all of these. Then touch a file, which is a trigger for generating a build, on the server.

I downloaded, and tested a lot of apps for all of this. Some were paid, most of them were free. Here’s what I chose for the workflow of publishing from the phone(iOS only for now). I’ll be reviewing the other apps that didn’t make it into this list, in separate series of posts (coming soon) on this blog.

1. Daedalus: I’ve written about Daedalus before. I love this app. I write all my posts, on the phone, in this app. It syncs to both my machines, in a folder on the Ulysses app. I love that app too. And it’s pretty great for writing in markdown. I use both Ulysses, and Daedalus to write - along with Desk. If I am travelling, it’s always Daedalus that I reach out for. I definitely recommend it.
2. Coda: I use Panic’s Coda for iOS to connect to the server, and upload the posts. It comes very well equipped with a lot of goodies like - an inbuilt editor that supports markdown, an ssh terminal, auto sync features for your projects, and a lot of connectivity options (FTP, FTPS, SFTP, WebDav, Dropbox, etc.). It’s a great, little arsenal of an app for any web developer out there, not just for someone looking to upload posts to his blog. Definitely worth the money.
3. Giphy: Giphy, the popular gif search engine, has an iOS app. Great for downloading GIFs files, and using them in posts. Along with Giphy, I use 9gag too. More often than not, I lose myself within 9Gag, so the usage is kept to a minimum.
4. GifX: An iOS app to make GIFs out of your videos, and photos. It’s pretty awesome. Again for generating GIFs for posts, etc.
5. Photo Compress: Compresses photos on the phone itself, before I upload them to the web server.

Considerations/Concerns:

• With this workflow, and setup in place, I still think that there’s still more room for improvement. For e.g. the build process takes about 8mins. ~~That’s a long time~~. So maybe, I can find a way to reduce that time. I am using some custom written plugins, and some other plugins to accomplish certain things. I think I may need to remove, or edit those to improve the overall build speed.
• The build process, as mentioned above, is a bash script that does everything. This may change later, if I find an easier way to build.
• For some reason, saving an edited post, which was edited directly on the server, from my home Mac machine, takes time. But editing the same post, from the iOS device, is quite fast. If nothing else, I look at it as making sure that the post(s) once uploaded, is as perfect as it can be. So, this may very well improve my editing skills.
• There’s also cause for concern w.r.t server security being compromised, at the least to some degree. It does take time to tighten things up.
• It’s not completely failsafe yet, and the build workflow I use can be more streamlined.
• I’ll come to know more, as I continue using this workflow to write, and publish daily on this blog. And whatever it is that find, w.r.t to this workflow, will eventually be ending up here on this blog.
• I may change over to Hugo (thank you Daniel Brinneman). Hugo is a GO based static site generator with no frills. And it’s fast. Though, I haven’t decided on that just yet. When I do get some time, I am going to experiment with it a little, before deciding.

Future:

Some more steps that I did like to include in this workflow, over the course of using it, are:

• Syncing all the files to both my local machines.
• New, and edited posts along with the media used in them, on these local machines would then be automatically synced with the master server. This would make it more easier to write, and edit a post when using either of the two ‘slaves’. Once automatically synced with the ‘master’, the ‘master’ would then start the build workflow.
• Increasing the server’s resources, and updating the code of the custom plugins would improve on the time it takes Jekyll to generate this blog.
• Upgrading to Jekyll 3. Better, and a tad bit faster.
• As mentioned above, maybe moving over to Hugo from Jekyll.

What’s Your Publishing Workflow Like? What Would You Change in My Workflow to Make It Better?

For now, this is about it. As always, I’ll be very interested to know how do you manage your publishing workflow? And I’ll be super interested to know if you can help make the workflow I use, better.

Just leave a comment, and let me know!

Note:

I am pretty aware that the entire build process could be accomplished with just a couple of bash scripts, and a cronjob. But, I do like Jenkins(maybe am more biased towards it), and would like to use it for some other projects on the server too.