Y Experiment's Articles

Setting up Sonatype

Both requirements for creating a new project in Sonatype repository are satisfied out-of-the-box if you have a public repository on BitBucket, GitHub, GitLab, or other.

SCM URL - publicly available repository with source code of your library - you need to be able to provide the URL pointing at this repository,

Project URL - publicly available homepage of the library - again, the URL is needed.

First of all, the OSSRH Guide walks you through the required process of setting up the account with Sonatype. It’s as simple as creating a Sonatype’s JIRA account and afterwards creating a New Project ticket. When creating the account try to use the same domain in your email address as the project is hosted on. I guess it makes it easier to validate the relationship with the groupId requested in the ticket. Creation of the New Project ticket is as simple as:

providing the name of the library in the ticket’s subject,

naming the groupId you want to use for distributing the library (make sure it is matching the root package of your code). Sonatype provides you with additional hints on choosing the right groupId for publishing your library in Choosing your coordinates guide.

providing the SCM and Project URLs to the source code and homepage of the library.

After creating yourself the Sonatype account (on their JIRA) you can login using the same credentials to the Nexus Repository Manager which is not required to be used in this guide, but can be used later to check on the published artifacts.

Notice that Sonatype advises that responding to the New Project ticket might take up to two business days, but in our case it was … 3 minutes. So don’t expect much delay before moving on to...

Setting up SBT

Following the OSSRH Guide, you now want to review the requirements of publishing to the central Sonatype repository. These are relatively detailed, but describe (in 2017) the configuration of Maven, which is only partially useful for SBT. Another not-very-helpful page of deployment with sbt provides some rope-ends but lacks useful information on how to proceed with sbt (hence this post). Sonatype’s recommendation here is to follow the Lightbend in sbt’s Manual - Using Sonatype, which indeed is helpful at this stage, but does not address the requirements (addresses only signing the package) and is missing information on the process to release (not just stage) the library.

The easiest path we’ve found to build a complete release process with sbt is as follows:

Install the GNU Privacy Guard and:

use it to generate the keypair you will use to sign your library,

publish your certificate to enable code signature verification,

make sure that the “gpg” command is in PATH (available to the sbt).

Once you have your public certificate install the sbt-pgp plugin in your user’s local sbt installation and tell it to use the installed GnuPG. In build.sbt:

useGpg := true

Install the sbt-sonatype plugin and configure it (notice that you can skip the sbt-pgp plugin as it is already installed). make sure to set the organization mapping in build.sbt to the groupId that you used in the New Project ticket. In build.sbt:

organization := "some groupId"

Install the sbt-release plugin. The configuration example provided in the sbt-sonatype plugin is pretty comprehensive but does not include opening of the repository, which results in the demo not working as (at least myself) expected. Fortunately this can be achieved by adding a ReleaseStep calling the sonatypeOpen command of the sbt-sonatype plugin.

Here is the modified example of changes to build.sbt that address this issue into consideration:

import sbtrelease.ReleasePlugin.autoImport.ReleaseTransformations._ releaseProcess := Seq[ReleaseStep]( checkSnapshotDependencies, inquireVersions, runClean, runTest, setReleaseVersion, commitReleaseVersion, tagRelease, ReleaseStep(action = Command.process("sonatypeOpen \"your groupId\" \"Some repository name\"", _)), ReleaseStep(action = Command.process("publishSigned", _)), setNextVersion, commitNextVersion, ReleaseStep(action = Command.process("sonatypeReleaseAll", _)), pushChanges )

Releasing

To get your project built, tested, tagged, signed, published, and pushed (all in one) execute:

sbt release

But... if things go awry you can try to manually replicate the publishing process (we’ll skip the building, testing, tagging, and pushing as obvious ones). There are many things that can go wrong, but you should be able to start debugging following these simple steps:

Open the staging repository:

sonatypeOpen "your groupId" "Some staging name"

Sign and publish to staging repository

publishSigned

Check the published artifact in the Nexus Repository Manager (same login as Sonatype Jira)

Close the staging repository and promote the release to central

sonatypeRelease

Payments Payments Payments!

Sharetribe’s documentation is quite good. Community is not large but enough to help you solve common problems. But there is one big problem - Payments.

PayPal payments are only available on marketplaces hosted at Sharetribe.com due to special permissions needed from PayPal. We hope to add support for PayPal payments to the open source version of Sharetribe in the future.

Read here

Snooping around you can find information that there are possibilities to set PayPal within your open source version of Sharetribe. In theory the open source version of Sharetribe has the same code as the Saas version. Yet there is no information regarding the integration with payment gateways.

There is no way to configure payments in sharetribe besides building your own e.g. using Stripe. Still, we highly valued the short time to market so we attempted to leverage what was there.

Everyone can run PayPal with sharetribe, but...

After carefully considering our options, we decided to try and configure our sharetribe instance with PayPal. And the “fun” began. This wasn’t easy, but could be achieved provided one important condition.

You will have to resign from taking commision from your sellers

Sharetribe uses reference transactions to take commissions from sellers. Description of such reference transaction can be found in the “legacy docs”. After closer inspection we learned that PayPal is resigning from this API and we didn’t manage to get a production access to this endpoint. Even if you will be able to run this feature in sandbox environment - I advise you to just skip this at all, as you will face difficulties enabling it in PayPal’s production environment. Eventually we to production skipping the settlement of commissions during payment process.

It has to be ok for you go without commission

When you are ok not taking commission during the payment process (e.g. can settle it in a separate process) then you can follow the steps below to configure PayPal with your development and next with your production environment.

These steps assume that you will use sharetribe as a platform for one marketplace, but in other cases you can easily modify SQLs according to your needs. This solution was tested with version 6.1.0. Steps are prepared to configure development version. Creating production is very similar.

Installing sharetribe and collecting needed data for configuration

Install Sharetribe following the sharetribe documentation

Run the application and create your first account - let’s call it admin account. We will use this account in our example as a admin and as a merchant (seller).

Create a PayPal account (or use your existing account). Go to paypal page and sign up.

Login to PayPal developer platform where you will find your test accounts

Next step is login to sandbox using the business account for which you have to set a password:

click profile row with business account

in tab profile click “Change password”

set the password and save

Login as a merchant in the sandbox environment to get Merchant account ID.

using your *-facilitator@* account login here

on setting look for merchant account ID

PayPal application ID for testing (development) purpose is defined globally: APP-80W284485P519543T - for your production you will need to get this from PayPal Customer service.

PayPal username, password and signature can be found in the PayPal developer platform

click on the business account (e.g. like *-facilitator@*) and then go to the API tab where you will find the credentials

Setting up PayPal configuration for platform

After collecting all data put them to config.yml

development:  paypal_username: [*facilitator_api1.*]  paypal_password: [password]  paypal_signature: [signature]  paypal_partner_id: [Merchant account ID]  paypal_app_id: APP-80W284485P519543T # Global test id  paypal_button_source: # Unique value given to you by your PayPal contact [optional]

Now it is time for some SQLs :). Connect to your Sharetribe database and run this, don't forget define variables.

# set PayPal gateway INSERT INTO payment_settings SET active = 1, community_id = 1, payment_gateway = 'paypal', payment_process = 'preauthorize', commission_from_seller = 0, minimum_price_cents = 400, minimum_transaction_fee_cents = 0, confirmation_after_days = 14, created_at = NOW(), updated_at = NOW(); # define variables SET @username=(SELECT username FROM people LIMIT 1); SET @owner_email=(SELECT e.address from people p left join emails e on (e.person_id=p.id) where p.username=@username); SET @merchant_account_id="[merchant_account_id]"; SET @paypal_username="[paypal_username]"; SET @paypal_app_id="APP-80W284485P519543T"; UPDATE people set is_admin=1 where username=@username; INSERT INTO paypal_accounts (person_id, community_id, email, payer_id, created_at, updated_at, active) VALUES (null, 1, @owner_email, @merchant_account_id, NOW(), NOW(), 1); SET @paypal_account_id=(SELECT LAST_INSERT_ID()); INSERT INTO billing_agreements (paypal_account_id, billing_agreement_id, created_at, updated_at, paypal_username_to, request_token) VALUES (@paypal_account_id, 'fake-agreement', NOW(), NOW(), @paypal_username, @paypal_app_id); INSERT INTO order_permissions (paypal_account_id, created_at, updated_at, request_token, paypal_username_to, scope, verification_code, onboarding_id, permissions_granted) VALUES (@paypal_account_id, NOW(), NOW(), 'fake-token', @paypal_username, null, null, null, 1); INSERT INTO transaction_processes (community_id, process, author_is_seller, created_at, updated_at) VALUES (1, 'preauthorize', 1, NOW(), NOW()); SET @transaction_process_id=(SELECT LAST_INSERT_ID()); UPDATE listing_shapes SET transaction_process_id=@transaction_process_id where name='selling'; #if you have listings added before you should modify transaction_process_id in listing entries too

Creating merchant payment configuration

Add new business account. This will be a merchant (seller) PayPal account

Connect PayPal merchant account with Sharetribe merchant account. Login with your Sharetribe admin account and go to Settings->Payments then click “Connect”

Don’t grant the permission to charge. This second step will require access to Reference Transaction API - and it’s unable to get it.

Make your first (sandbox) payment

Login to sharetribe with your admin account (or other configured and connected to the PayPal account).

Add a new listing and logout.

Create a buyer account.

Remember to set the password for buyer account in PayPal developer platform

Logged in as a buyer - click buy on listing - and then process it through PayPal payment.

Login as a merchant (in our case - mechant is an admin account) and confirm the transaction. Go to the inbox (envelope icon) select transaction and accept the payment.

Login to the sandbox with your PayPal buyer account to see the transactions - just to be sure everything was done correctly.

Production ready

Going to production is relatively easy when you know how this works from the  development perspective. The main difference is that you are working with live accounts and you will need exchange few emails with PayPal.

At first you need to create the business account (the target one you will use to make business). Then you have to ask PayPal to enable to the API access for it. Then you will have almost all of the needed configuration parameters.

The last missing piece of information is the PayPal application id. To get such ID you have to fill the big form here: https://www.paypal-apps.com.

All the collected data has to be put into your production configuration file and followed with the execution of the same SQLs (same as for the development) on your production database. And voila! You have sharetribe with PayPal payments.

Conclusion

A few words on bitbucket pipeline

You can use Pipeline for free (you have available 50 minutes running time per developer in project per month) that is why this is good solution for small quick projects. Configuration of CI is very easy and based on Docker. Actually you can use only one docker container, but plans are that user is able to use more than one container in one build - which gives a lot of possibilities.

Atlassian have prepared a few of docker images, that allow you to run the test for different languages with different databases. But in this catalogue there is no Scala image. That is why I decided to describe how to build pipeline with Scala language.

Configure your Pipeline

Scala (Sbt) container

The first what you need to have is a container where you can build your scala application. Testing your CI pushing your code to bitbucket on every change can be very hard. That is why you are able to use container locally and test on it if everything work well. To open you locall pipeline you need to run this command:

$ docker run -it --volume=$(pwd):/project --workdir="/project" --memory=4g --memory-swap=4g --entrypoint=/bin/bash hseeberger/scala-sbt

Note: this can modify ownership of files on your local storage. To solve problem clean generated files.

$ sbt clean clean-files

Run your test to be sure is everything is configured.

$ sbt test

First build

When your tests are working on docker container well, then we can move it this to our new CI. To be able to run tests in pipeline you need to enable pipeline before. Go to your project page and click Pipeline link.

When your pipeline in project will be enabled then create bitbucket-pipelines.yml file in your root project directory.

image: hseeberger/scala-sbt pipelines: default: - step: script: - sbt test

Bitbucket will discover your bitbucket-pipelines.yml file after push and run your test.

CI is not enough. We want more!

It’s convenient when you don’t have to run your deployments manually. That is why we are going to configure CD now. This will be scenario when we have server with ssh access. To be able to connect to our servers we need to store our private key. We don’t want to have that data in repository - and solution comes with pipeline.

To add your private key to pipeline and allow your docker to connect to server follow steps in piepline documentation. After configuring pipeline and server my bitbucket-pipelines.yml looks that:

image: hseeberger/scala-sbt pipelines: default: - step: script: - sbt test branches: production: - step: script: - echo "Run tests" - sbt test - echo "Prepare ssh connection" - mkdir -p ~/.ssh - cat conf/ci-scripts/deploy_known_hosts >> ~/.ssh/known_hosts - (umask 077 ; echo $DEPLOY_SSH_KEY | base64 --decode > ~/.ssh/id_rsa) - echo "Prepare deployment variables" - SSH_COMMAND='ssh [user]@[domain]' - DEPLOYMENT_PATH=/home/project/deployments/alpha - DEPLOYMENT_TARGET_STAGE=target/universal/stage - APPLICATION_NAME=application - echo "Run deployment" - $SSH_COMMAND "cd $DEPLOYMENT_PATH && git fetch --all && git reset --hard HEAD && git checkout $BITBUCKET_COMMIT" - $SSH_COMMAND "cd $DEPLOYMENT_PATH && sbt stage" - $SSH_COMMAND "kill $(cat $DEPLOYMENT_PATH/$DEPLOYMENT_TARGET_STAGE/RUNNING_PID)" - $SSH_COMMAND "$DEPLOYMENT_PATH/$DEPLOYMENT_TARGET_STAGE/bin/$APPLICATION_NAME -Dconfig.resource=application.prod.conf < /dev/null > /dev/null 2>&1 &"

To deploy code you need to run

$ git push origin branch-to-deploy:production

You can consider deploy your code from tags rather than from branch. Then you need to modify your bitbucker-piepelines.yml. Remove section branches and add:

image: hseeberger/scala-sbt pipelines: // [...] tags: release-*: - step: script: // [...] test steps // [...] deployment steps

I want to have release button

And thanks to pipeline you can have it. The only what you need to do is to configure your bitbucket-pipelines.yml

pipelines: default: - step: script: - sbt test custom: deploy-to-production: - step: script: // [...] deployment steps

Look to piepeline manual for more

Conclusion

I like it how bitbucket pipelines can be managed. It’s simple and easy tool, but can be very powerful.

New features are still being developed and in my opinion pipeline can be very strong competitor for Travis.

Resouces:

https://bitbucket.org/product/pricing/upcoming

https://confluence.atlassian.com/bitbucket/bitbucket-pipelines-792496469.html

https://confluence.atlassian.com/bitbucket/access-remote-hosts-via-ssh-847452940.html

https://github.com/hseeberger/scala-sbt

https://confluence.atlassian.com/bitbucket/language-guides-for-bitbucket-pipelines-856821477.html