#travis-ci

I am so happy it works I will definitely try to implement CI any time I can.

#HuazhuGroup #Hacking #Breach

Huazhu Group customers’ data breach is now being offered for sale on Dark Web for 8 bitcoin

IT is going to happen, and it is happening as this piece was being written, and will continue to happen as cyber bad guys seem to always be a step ahead of organizations’ cyber-security sentinels.

What is being referred here is “hacking,” the breaching by cyber attackers of security protocols of…

Continuous integration allows the whole build, test, and release pipeline to be automated so you have more time to code / sleep. I recently started using it in all my open source projects and for a while I couldn’t seem to find a way to get both linux and mac integration working properly for Python. I managed to look through a bunch of other peoples .yml files and find an answer.

We will be using Travis-ci for this, this ci system uses yml files to set everything up, The file must be called ".travis.yml" and placed in the root of your project. yml files are weird and dont support tabbed indents so we will be using spaces instead.

We first need to specify our language and what operating systems we are going to use. We must use a matrix to specify that for the osx version we are using objective-c instead of python. For some reason if you choose python as the language for osx it fails to install python

language: python matrix: include: - os: linux language: python python: 3.5 - os: osx language: objective-c python: 3.5

So now that we've got it starting linux and osx we need to make sure python gets installed on osx. I used some simple logic to differentiate the linux and osx instructions because they varied somewhat.

before_install: - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update ;fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew install python3 ;fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then python3 -V ;fi

For osx we need to use brew to install python3 or whatever version you're using. I used if statements so when the linux version runs through it doesnt do the osx steps.

Now we make our install logic, if you've packaged your code properly you can just use your setup.py

install: - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then python setup.py build install ;fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then python3 setup.py build install;fi

Its important to note that the osx has to call python3 specificy if you installed the python 3.x package.

Now we can run our test code once the installation is finished.

script: - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then python setup.py test ;fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then python3 setup.py test ;fi

Now if you want you can make Travis-ci deploy your project to whatever package manager you want. In this I use pypi (pip). Now to do this on an open source project on github you should probably get your password hashed before putting it on the web. Its hashed with a specific salt linked to your account specifically so others cant deploy your code.

Learn how to secure your password here: https://docs.travis-ci.com/user/encryption-keys/

deploy: provider: pypi user: MyUserName distributions: sdist skip_cleanup: true password: secure: MyUniquePasswordHashed on: branch: master tags: true

For the distribution I used sdist so it uploads the source code to pypi instead of bdist which builds for the current os. when you download the package with pip it will build it for you so dont worry.

I set the target branch to master so it grabs from that branch and set tags to true. I did that so it only builds when you tag the commit and on github that means you made a release. Travis-ci will still build and test your code if you dont tag the commit and you can still tag commits on other branches without it deploying.

When we use Travis CI or other continuous integration tools, we start looking at automation, notifications and testing as part of our daily tasks. After a short period, you’ll surely start to wonder: “How did I survive without this process before?” Basically devs love CI tools because they can sleep better at night eliminating hassles often caused by long integration cycles: broken builds, manual merging hell and regressions in a huge fragmented world.

Adding Monkop CLI after your build process will provide your team a dedicated bot that automatically installs your app on all relevant real devices (including different OS versions / screen sizes) and starts playing with it, looking for crashes and other common problems.

Travis-ci는 저번에 블로그에서 한번 언급한 적이 있다.

필자는 기존에 진행중이던 WPF 프로젝트를 travis-ci를 통해 build status를 Github와 연동하고 싶었으며, 실제로 행동에 옮겼었다. 

하지만 결과는 `Controls' does not exist in the namespace `System.Windows' 와 같은 에러로그로 뒤덮였다.

Travis-ci를 들어본적이 있는가?

안드로이드 개발자에게는 젠킨스라는 단어가 더 익숙할 것이다. travis-ci의 ci는 Continuous Integration 한국말로 지속적인 통합을 의미하는데, 사실 말이 복잡해서 그렇지 도입이유는 통합단계에서 발생하는 장애를 개선하기 위해서다.

I came across a question recently in StackOverflow:

I would like to use Travis CI for my open-source project. The issue that Travis doesn't provide any ways to publish produced artifacts (though, they have this in their future plans).

What are workarounds to publish/upload artifacts somewhere? I'm allowed to execute any scripts on a CI machine.

Simple upload will work, but there is security issue: anyone will be able to upload something in the same way as all sources are public.

It was asked in 2012, and while a lot has changed since then, none of the answers recommended Bintray, which I think is a real shame, as it’s a repository system designed for this stuff.

I added my answer to the question, and thought I could expand more on the solution here.

Lets go over the basics:  you have an open source project hosted in github, and you want to tag releases, and have those releases build and make their way out to JCenter (bintray’s popular repository, and The Easy Way into Maven Central) for others to consume/use.

If you are using Java & Maven, this post is for you.  If you are using Java & <insert better tool here> then most of this post still applies, but you’ll want to do different deployment steps with your given tool.  Gradle is quite popular (and arguably better than Maven), and I’ve heard that process is easier, as there are better ways to do it with gradle.  Since my project was with maven, I’ll focus on that here.  Lets get started.

Step 1 - Create your accounts

You need accounts for Bintray and Travis-CI.  I’m going to assume you don’t have these, if you do, skip this and go to Step 2.

1.1 Travis-CI Setup

Head to https://travis-ci.org/ and create an account.  Link it to your github account, and give it access.  It should show a list of code repos in github that you control.  Select the repo you want, and enable travis-ci for that repo.

There’s more to it, so at this point, take a moment and read the getting started guide: http://docs.travis-ci.com/user/getting-started/

One last thing, before proceeding, read the section from travis-ci about encrypting secret variables (environment variables), we will do this in the next step so that when travis runs, maven has access to a username and password for the bintray publishing step:  http://docs.travis-ci.com/user/encryption-keys/

1.2 Bintray Account

You’ll need to create a Bintray account, and also create an API key that is associated with your account.

Go to https://bintray.com/ and create an account.  It helps to also register/link the account to your github account.  Once that’s done, head to your profile, and create an API Key that you’ll use later with maven (you reference an encrypted version of this as an environment variable in a custom settings.xml we create for maven to release with).

Next, create a maven repo for your user account (or bintray organization if you set one of those up).

Lastly, create a package in bintray that represents your project’s artifacts (published stuff goes in a “package”, each release is versioned and good to go).  https://bintray.com/howbintrayworks

During the package creation, you can also link it to JCenter, which will expose it to a larger public audience and let others more easily download and use it as a maven dependency.

Step 2 - Setup Maven for Bintray

Edit your pom.xml to include a distributionManagement section.  Looking at the <url>:  note that you need to fill in your bintray account/username, the repo name, and the project name.  The “maven” part stays the same, this is following the same examples that bintray provides:

<distributionManagement> <repository> <id>my-bintray-id</id> <url>https://api.bintray.com/maven/myUserName/myRepoName/my_awesome_project;publish=1</url> </repository> </distributionManagement>

Next, setup a custom maven settings.xml file that has your bintray username and bintray API key.  Note that you will first need to encrypt the BINTRAY_USER and BINTRAY_API_KEY variables using travis, (see this guide: http://docs.travis-ci.com/user/encryption-keys/).  Here’s an example settings.xml:

 <servers>    <server>      <id>my-bintray-id</id>      <username>${env.BINTRAY_USER}</username>      <password>${env.BINTRAY_API_KEY}</password>    </server>  </servers>

When travis runs, maven will have access to the decrypted environment variables.

Lastly, for maven, we need to add the release plugin to our pom.xml, so that we can do our releases (git tag, bump versions, make commits, etc):

 <build>    <plugins>      <plugin>       <groupId>org.apache.maven.plugins</groupId>        <artifactId>maven-release-plugin</artifactId>        <version>2.5.2</version>      </plugin> </plugins> </build>

Step 3 - Configure .travis.yml

Next, setup your .travis.yml yaml configuration file.  This file sets steps and configurations needed for Travis to run.  There are many guides for this, read these first before writing the .travis.yml file:

http://docs.travis-ci.com/user/languages/java/

http://docs.travis-ci.com/user/travis-lint/

http://docs.travis-ci.com/user/ci-environment/#Environment-variables

One important note, is that Travis provides an “after_success” section you can add that will execute after every successful build.  This is key for us, as we will add checks in this so that after every success we check for:

A git tag.

A specific JDK version (whatever version you want to publish from).  This lets you build under multiple JDKs but publish under one.

Anything else we desire.

Given those checks pass, we run “mvn deploy” to publish our built artifacts.

Here’s an example .travis.yml:

language: java jdk:  - oraclejdk7  - oraclejdk8 after_success:  - mvn clean cobertura:cobertura coveralls:report javadoc:jar  - test "${TRAVIS_PULL_REQUEST}" == "false" && test “${TRAVIS_JDK_VERSION}” == “oraclejdk7″ && test "${TRAVIS_TAG}" != "" && mvn deploy --settings travis/travis-settings.xml branches:  only:    - master      # Build tags that match this regex in addition to building the master branch.    - /^my_awesome_project-[0-9]+\.[0-9]+\.[0-9]+/ env:  global:    - secure: cfHTvABEszX79Dhj+u8/3EahMKKpAA2cqh7s3JACtVt5HMEXkkPbeAFlnywO+g4p2kVENcQGbZCiuz2FYBtN3KrIwFQabJE8FtpF57nswPRrmpRL+tWcYtipVC2Mnb4D7o6UR2PiC7g20/    - secure: cfHTvABEszX79Dhj+u8/3EahMKKpAA2cqh7s3JACtVt5HMEXkkPbeAFlnywO+g4p2kVENcQGbZCiuz2FYBtN3KrIwFQabJE8FtpF57nswPRrmpRL+tWcYtipVC2Mnb4D7o6UR2PiC7g20/

Notice that in the “after_success” part, I’m calling out to some other maven targets, and then testing for our conditions:  not a pull request, has a tag, and finally doing the mvn deploy (passing in my travis-settings.xml file from step 2 above).

Also notice that I white-list the build to only build on master branch, and any tag that matches the regex for “my_awesome_project”.

Step 4 - Make a Release

Finally, lets make a release.  For this, we use the maven release plugin from our local dev box.

Make sure you are caught up with #master branch (git fetch/pull).

Run mvn release:clean and mvn clean.

Now, run the maven release:

mvn release:prepare 

This will bump our version in the pom, removing -SNAPSHOT, tag it with the version we say to (use the default, so the regex in travis matches), push commits/tags, and then bump the version and add -SNAPSHOT, making a final commit so the latest version is in git.

Ultimately travis sees this tag, and our after_success tests pass, and a ‘mvn deploy’ occurs, which, with the settings.xml, sends the artifacts into Travis at our repo and package we set up.

After every release you run, do a local mvn release:clean, and git fetch/pull.

Final Notes:  The big end to end system

What do we get with this?  We get a system that looks like this: