Develop, test and deploy standalone apps on CloudBees

CloudBees is a cloud platform providing repository, CI service (Jenkins) and server for your apps. So everything you need to develop, test and deploy. There are many options, e.g. repository can be Git or SVN, for server you can choose Jetty, Tomcat, Glassfish, JBoss, Wildfly etc. It is also possible to run standalone applications, which are provided with port number, so you can start your own server. And that’s the case we’ll cover here.

spray.io is Scala framework for web apps. It allows you to create standalone web-apps (starting their own server, spray-can) or somewhat limited .war ones (spray-servlet), which you can deploy on JEE server like Glassfish, JBoss etc. We are going to use standalone here.

You can clone the app from github Let’s take a quick look at it now.

The app

Boot

The Boot file is Scala App, so it’s like java class with main method. It’s runnable. It creates Service actor, which is handling all the HTTP requests. It also reads port number from app.port system property and binds the service to the host and port. app.port is provided by CloudBees, if you want to run the app locally, you need to set it e.g. by jvm command line -Dapp.port=8080.

Service

Service has MyService trait, which handles routing to empty path only. Yes, the app is not very complicated ;)

Buildfile

build.gradle file is a bit more interesting. Let’s start from it’s end.

• mainClassName attribute is set to Scala App. This is the class that is going to be run when you run it locally from command line by gradlew run.
• applicationDefaultJvmArgs is set to -Dapp.port=8080 and it’s also necessery for running locally from gradle. This way we set port which Service is going to be bound to.
• jar.archiveName is a setting used to set generated .jar name. Without it it’s dependent on the project directory name.

You can run the application by issuing gradlew run (make sure gradlew file is executable). When it’s running, you can point your browser to http://localhost:8080 and you should see “Say hello to spray-routing on spray-can!” Nothing fancy, sorry.

There is also “cb” task definde for gradle. If you issue gradlew cb, it builds zip file, with all the dependency .jars, and szjug-sprayapp-1.0.jar in it’s root. This layout is necessary for CloudBees stand alone apps.

Deploy to CloudBees

First you need to create an account on CloudBees. If you have one, download CloudBees SDK - so you can run commands from your command line. On Mac, I prefer brew install, but you are free to choose your way.

When installed, run bees command. When run for the first time, it asks your login/password, so you don’t need to provide it every time you want to use bees.

Build .zip we’ll deploy to the cloud. Go into the app directory (szjug-sprayapp) and issue gradlew cb command. This command not only creates the .zip file, it also prints .jars list useful to pass to bees command as classpath.

Deploy the application with the following command run from szjug-sprayapp directory:

bees app:deploy -a spray-can -t java -R class=pl.szjug.sprayapp.Boot -R classpath=spray-can-1.3.1.jar:spray-routing-1.3.1.jar:spray-testkit-1.3.1.jar:akka-actor_2.10-2.3.2.jar:spray-io-1.3.1.jar:spray-http-1.3.1.jar:spray-util-1.3.1.jar:scala-library-2.10.3.jar:spray-httpx-1.3.1.jar:shapeless_2.10-1.2.4.jar:akka-testkit_2.10-2.3.0.jar:config-1.2.0.jar:parboiled-scala_2.10-1.1.6.jar:mimepull-1.9.4.jar:parboiled-core-1.1.6.jar:szjug-sprayapp-1.0.jar build/distributions/szjug-sprayapp-1.0.zip

And here abbreviated version for readability:

bees app:deploy -a spray-can -t java -R class=pl.szjug.sprayapp.Boot -R classpath=...:szjug-sprayapp-1.0.jar build/distributions/szjug-sprayapp-1.0.zip

spray-can is an application name, -t java is application type. -R are CloudBees properties, like class to run and classpath to use. Files for classpath are helpfully printed when gradle runs cb task, so you just need to copy & paste.

And that’s it! Our application is running on the CloudBees server. It’s accessible at the URL from CloudBees console.

Use CloudBees services

The app is deployed on CloudBees, but is that all? As I mentioned we could also use git repository and Jenkins. Let’s do it now.

Repository (Git)

Create new git repository on your CloudBees account. Choose “Repos” on the left, “Add Repository”… it’s all pretty straightforward.

Name it “szjug-app-repo” and remember it should be Git.

Next add this repository as remote one to your local git repo. On the repositories page on your CloudBees console there is very helpful cheetsheet about how to do it.

First add git remote repository. Let’s name it cb

git remote add cb ssh://git@git.cloudbees.com/pawelstawicki/szjug-app-repo.git

Then push your commits there:

git push cb master

Now you have your code on CloudBees.

CI build server (Jenkins)

It’s time to configure the app build on CI server. Go to “Builds”. This is where Jenkins lives. Create new “free-style” job.

Set your git repository to the job, so that Jenkins checks out always fresh code version. You’ll need the repository URL. You can take it from “Repos” page.

Set the URL here:

Next thing to set up is gradle task. Add next build step of type “Invoke gradle script”. Select “Use Gradle Wrapper” - this way you can use gradle version provided with the project. Set “cb” as the gradle task to run.

Well, that’s all you need to have the app built. But we want to deploy it, don’t we? Add post-build action “Deploy applications”. Enter Application ID (spray-can in our case, region should change automatically). This way we tell Jenkins where to deploy. It also needs to know what to deploy. Enter build/distributions/szjug-app-job-*.zip as “Application file”.

Because you deployed the application earlier from the command line, settings like application type, main class, classpath etc. are already there and you don’t need to provide it again.

It might also be useful to keep the zip file from each build, so we can archive it. Just add post-build action “Archive the artifacts” and set the same zip file.

Ok, that’s all for build configuration on Jenkins. Now you can hit “Build now” link and the build should be added to the queue. When it is finished, you can see the logs, status etc. But what’s more important, the application should be deployed and accessible to the whole world. You can now change something in it, hit “Build now” and after it’s finished, check if the changes are applied.

Tests

Probably you also noticed there is a test attached. You can run it by gradlew test. It’s specs2 test, with trait MyService so we have access to myRoute, and Specs2RouteTest so we have access to spray.io testing facilities.

@RunWith(classOf[JUnitRunner]) is necessary to run tests in gradle.

Now when we have tests, we’d like to see tests results. That’s another post-build step in Jenkins. Press “Add post-build action” -> “Publish JUnit test result report”.

Gradle doesn’t put test results where maven does, so you’ll need to specify the location of report files.

When it’s done, next build should show test results.

Trigger build job

You now have build job able to build, test and deploy the application. However, this build is going to run only when you run it by hand. Let’s make it run every day, and after every change pushed to the repository.

Summary

So now you have everything necessary to develop an app. Git repository, continous integration build system, and infrastructure to deploy the app to (actually, also continously).

Think of your own app, and… happy devopsing ;)

Mercurial installation and remote server interaction by hg-login

I want to have mercurial on my machine, but I also want to have "central repository" for continuous integration builds. I chosen mercurial because it is fast (like git), and it has good support in NetBeans (at least I heard it has ;)).

To push/pull changes to/from server, I use hg-login. Ssh is also necessary. Ok, so here are the steps:

1. Install mercurial on both client and server.

apt-get install mercurial

2. Create user mercurial on the server, and disable his shell.

sudo adduser mercurial

Edit /etc/shadow and set his password to "*".

3. Go to server (ssh will suffice) and switch to user mercurial.

sudo su - mercurial

4. Go to /home/mercurial and create repositories directory.

cd /home/mercurial
mkdir repositories

5. Create file hg-login in /home/mercurial. Copy script there. Script can be found on http://mercurial.selenic.com/wiki/HgLogin

mcedit hg-login

It can be necessary to change the script. In my case I needed to change /usr/local/bin/hg to /usr/bin/hg. Change to where your hg command is located.

6. Go back to client machine. Generate ssh keys pair for yourself.

ssh-keygen

Add private key to your ssh agent.

ssh-add

This way when you connect by ssh from client, you are automatically authenticated.

7. Go to server again. In /home/mercurial/.ssh create file authorized_keys.

cd /home/mercurial/.ssh
touch authorized_keys

In this file put line:

command="/home/mercurial/hg-login franek",no-port-forwarding,no-X11-forwarding,no-agent-forwarding [key]

franek is user name. Replace [key] with content of your .ssh/id_rsa.pub file from client. Put whole content there (with ssh-rsa in the beginning and <user>@<host> at the end).

8. Create repository in /home/mercurial/repositories/.

cd /home/mercurial/repositories
mkdir myapplication
cd myapplication
hg init

9. Allow franek to access myapplication repository. In /home/mercurial/repositories create file myapplication.allow. In this file list users who has to have access to myapplication repository. In this case franek, so the file looks like this:

franek

10. On client machine, go to directory where your application is: cd <yourdir>/myapplication

11. Create local mercurial repository.

hg init

12. Add all what is in /myapplication directory to local repository.

hg add

13. Commit added files.

hg commit

You'll need to edit commit comment.

14. Push changes to remote repository.

hg push ssh://mercurial@yourserver/myapplication

While creating this recipe, hg-login site was very helpful.

As I mentioned before, I chosen mercurial because of Netbeans. But I don't find Netbeans support of mercurial satisfactory. I am used to eclipse and it's great support for CVS. In eclipse, if there are some changes in CVS and I don't have them (incoming changes), I can see what has changed (file by file diff) before I update my code. I failed to do this in Netbeans with mercurial.

Setting up backup by rsync

I want to backup my files on another server. So I go to another server, and create file /etc/rsyncd.conf:

secrets file = /etc/rsyncd.secrets
#Global properties
read only = yes
list = yes
#User on server
uid = backup
#User's group
gid = backup

#Protected share for backups.
[files]
comment = For your eyes only
path = /home/backup/
auth users = franek
read only = no
hosts allow = 192.168.1.64
hosts deny = *
list = yes

We are using user backup as rsync files owner. Don't forget to create him.

So now we need to create entry for franek in rsyncd.secrets:

franek:his_password

rsyncd.secrets cannot be readable for all. In such case rsync daemon will not allow access to protected shares. Set permissions to 600.

We need to do one more thing before starting the daemon. Go to /etc/default/rsync and set RSYNC_ENABLE=true

Ok, issue sudo /etc/init.d/rsync start and rsync deamon is running.

Now go to the client machine. The one we want to make backups from. I want to backup whole /home, so my command looks like this:

sudo rsync -aXAvz --delete --delete-excluded --exclude-from=$DIR/backup.excludes --password-file=$DIR/rsync-pass /home franek@moon::files

So there is sudo rsync, and then some options:

-a the same as -rlptgoD, which is:
-r recurse into directories
-l copy symlinks as symlinks
-p preserve permissions
-t preserve modification times
-g preserve group
-o preserve owner
-D preserve device files, preserve special files
-X preserve extended attributes
-A preserve ACLs
-v be verbose
-z use compression

Then there is --delete and --delete-excluded. It means if there is some file on copy, but there is no such file on source, delete file from copy. --delete-excluded means delete all excluded files from copy.

I keep exclusions patterns in separate file. Format is quite simple:

lost+found
*/.Trash/
*/.thumbnails/
cache/
Cache/
.Cache/
.cache/

My share is password protected. Only user franek can access it, and he needs to give his password. If you want to do it by system (e.g. cron), you don't have the possibility to type the password. However, you can put it into file and make file readable only for user (permissions 700). Then you can just point to that file with --password-file option.

Next there is just source and destination. Just like in cp or smb. Source is pretty simple in my example. Destination is a bit more complicated: <user>@<server>::<share-name>

Setting samba on ubuntu server for sharing files

I am building home server, so now I want to configure it in a way so I could put photos there, and everyone could access them.

I assume samba is already initialized.

Guest access
We want everybody to be able to store files there, and everybody to be able to read them. So let's give guest access to it. But somebody has to be the owner of the files, right? (In terms of Unix file ownership, chown). So we neet to tell who guest user maps to. Lets call our Unix user henry:

sudo adduser henry

Set some password too.

Now, let's tell samba that guest should map to henry. In /etc/samba/smb.conf in [global] section:

guest account = henry

We are ready to create share for everybody now. Again, in /etc/samba/smb.conf:

[for-all]
path = /home/henry
browsable = yes
read only = no
guest ok = yes
create mask = 644

Now we created share called "for-all", it is stored at /home/henry, and everybody can read and write it. Files put to server into for-all share are going to be read-write for owner (henry), and read for everyone. However, it is not so important as everybody can access them like owner through samba.

Users access
We can also make it available only for users. Let's say we want samba user henry:

sudo smbpasswd -a henry

Set password. User created.

Now we need to disable guest access:

guest ok = no

Now we can connect from another machine by smbclient:

smbclient //192.168.1.1/for-all -U henry

Of course put your server's IP instead of 192.168.1.1

Mount as fs for users manually
You can mount samba share to some directory on your client machine. Just like some filesystem. Install smbfs package:

sudo apt-get install smbfs

If you have sudo access, you can mount samba share. First, create directory to mount it:

mkdir smb-share

Now, mount share:

sudo mount -t cifs //192.168.1.1/for-all smb-share -o username=henry,noexec,uid=local_user,gid=local_group

noexec means you'll be not able to run any executable files from that share. uid and gid will render all files in mounted share to belong to user local_user and group local_group on client.

If you want more users to be able to mount the share, you can create group for them:

sudo addgroup samba

Now let's give this group rights to mount remote samba shares. Run:

sudo visudo

And in the group section add following line:

%samba   ALL=(ALL) /bin/mount,/bin/umount,/sbin/mount.cifs,/sbin/umount.cifs

Add user frank to that group:

sudo adduser frank samba

From now on, frank can mount samba shares.

Configure postfix to send emails for root to your account

Goal: make our small ubuntu server forward email which has "to: root" to your private mail.

Install postfix.

sudo apt-get install postfix

Some console app for sending emails can be handy too:

sudo apt-get install mutt

Edit /etc/postfix/main.cf. In mydestination enter host, host.domain, localhost.domain, and also empty domain:

mydestination = serverek.przepompownia.com, localhost.przepompownia.com, , localhost

Restart postfix. I'm not sure if it is really needed, but won't hurt ;)

sudo /etc/init.d/postfix restart

Ok, now edit /etc/aliases. Add something like this:

root: franek
franek: mickey@my.mail.com

Dont forget to run:

sudo newaliases

This way everything sent to root is forwarded to franek, and everything sent to franek is forwarded to mickey@my.mail.com

Now you can try sending some mail to "root". You can use mutt for this.
UPDATE:
Recently my ISP (neostrada, TPSA) blocked outgoing connections to port 25 (standard port for connecting with SMTP servers). What to do to send emails to port 587 is well described here. Author only forgot to add that you also need this:

transport_maps = hash:/etc/postfix/transport

in your main.cf