#spark-jobserver

#Scala #akka @ApacheSpark

Python job support! (mostly @mattinbits, some @CalebFenton)

Some settings like job-jar-paths renamed to job-bin-paths (@nikkiSproutling)

New Scala API! The expanded API includes passing in a JobEnvironment which provides access to job ID, context information; it also offers more type safety in job return type, validation etc. (@velvia, others)

Spark 2.0 support (see spark-2.0-preview branch) (@f1yegor, @addisonj)

Changed minor HTTP responses to be all JSON (@roarking)

Deprecate JobFileDAO. H2 is now the default. Furthermore, H2 and JobFileDAO cannot be used with context-per-jvm. (@noorul)

Context creation API (POST /contexts) now takes a config in the POST body! (@casey-green)

Make create job API to return more information (@noorul)

Upgrade Slick to 3.1.1 (@hntd187)

Spark driver cluster mode for mesos (@LeightonWong)

Fix broken links to ec2 scripts, #369 (@noorul)

EC2 VPC deploy fixes (@mcnels)

Only set spark.executor.uri if env var is set (@addisonj)

Add Scala version to Docker image (@mariussoutier)

Integrate converage into CI using codecov.io service (@hntd187)

Remove akka dependency from api module (@f1yegor)

Eliminate POST and DEL /job race conditions (@addisonj)

Improve Chinese document (@stanzhai)

Return error if data file can't be deleted (@CalebFenton)

Make dbcp optional, default: disabled (@noorul)

Fix for logging issue in dev mode, #475 (@noorul)

Fix flaky tests (@TimMaltGermany)

Increase size of config/input that can be submitted via custom Akka serializer (@rmettu-rms)

Update build plugins (@hntd187)

README fix (@Vincibean)

Ensure Scala compiler dependency has correct version (@@aganim-chariot)

Docs for YARN queue config option (@ash211)

Make JMX port configurable (@casey-green)

Update test description, PR #481 (@oranda)

Per-user authenticated contexts, PR #469 (@TimMaltGermany)

Fix for Delete file API issue, #507 (@noorul)

Fix UI not showing running jobs if completed jobs fills the limit, #547 (@TianLangStudio)

Fix Jar name issue on Windows (@TianLangStudio)

Forked JVM processes must have their own JobDAO, #353, First step towards HA (@noorul)

Change cluster status of removed contexts to down (@derSascha)

Include Python exception stacktrace on failure (@CalebFenton)

UI - Use relative paths so it works if running in a context-path (@sjoerdmulder)

Bash style fixes (@sks)

EC2 deployment script fixes (@mcnels1)

Python Job API - Format standard output from process (@windelinckx)

Fix test issues on Windows (@doernemt)

Use OpenJDK for Docker base image (@xjodoin)

Cleanup LDAP authentication and add filter config. (@derSascha)

Giter8 template available at https://github.com/spark-jobserver/spark-jobserver.g8 (@noorul)

Delete binary API (@f1yegor)

Fix start-manager.sh for Docker environments (@fmcgough)

#Scala #akka @ApacheSpark

1 context per JVM. This forks off a separate process for each SparkContext, and is a major change in architecture. (EXPERIMENTAL) (@velvia, @Ankit1010, many others)

Update to Spark 1.6.0 (@RaafatAkkad, @apivovarov)

NamedObjects, an abstraction of NamedRDDs to other types of things (@TimMaltGermany / KNIME)

NamedBroadcasts (@CBribiescas)

Per-job dependent jars (@amarouni)

DEBUG_PORT env var allows remote debugging (@addisonj)

UI links to status/config (@h0ke)

Fix bug - invalid error w no user/pass #329 (@slater-ben)

Chunked stream encoding for Stream[_] job output (@leone17)

Workaround - custom error classes for context-per-jvm (@ishassan)

Spark Hadoop configuration in spark.context-settings.hadoop (@hntd187)

Support user impersonation for an already Kerberos authenticated user (@mateilucianstefan)

Expose MAXDIRECTMEMORY to adjust direct memory buffer size (@addisonj)

Expose JAVA_VERSION for building with say JDK8 (@stannie42)

Fix docker build (@stannie42)

Redirect /dev/null to stdin in server_start.sh (@apivovarov)

Database migration scripts and better table definitions (@v-gerasimov)

Chinese doc updates (@zp-j, @liujuncheng)

Minor improvements in style (@apivovarov)

SBT and other dependency/plugin upgrades, including to Mesos 0.28 (@hntd187, @jaceklaskowski)

README change to document overriding default meta data DB backend (@noorul)

README typo fix (@qayshp)

#Scala #akka @ApacheSpark

New route POST /contexts?reset=reboot to close and restart all contexts defined in config (@wildwood)

CORS support and EC2 deploy scripts (@David-Durst)

Updated to Spark 1.5.2 (@zeitos, @apivovarov)

Updated Docker container, much smaller (based on jre-java7) with configurable Mesos version

Return job status uniformly for both /jobs and /jobs/ routes (@noorul)

Fix issue #247: Allow non-computed named RDDs (@TimMaltGermany / KNIME)

Fix for PostgreSQL varchar issue (@ffarozan)

Update Yarn docs, set --master yarn-client in start script (@lant)

Update Spray to 1.3.3 (@v-gerasimov)

Allow Web API timeouts to be configurable via short-timeout (@v-gerasimov)

Chinese documentation at doc/chinese/ ! (@linrunzhang)

#Scala #akka @ApacheSpark

Default to Spark 1.4.1 (@zeitos)

LDAP Authentication support via Apache Shiro (@TimMaltGermany / KNIME)

HTTPS Support! (@koettert @TimMaltGermany / KNIME)

Load jars automatically on startup (@lossyrob / Azavea)

/data routes (The KNIME team)

JavaSparkJob

Fix server-stop.sh so it works with spark-submit (@addisonj @RussellSpitzer @jmelching)

Upgrade to spray-json 1.3.2 (@v-gerasimov)

Upgrade joda-time to 2.2 (@jwalgran)

Docker fixes, such as for logging (@MeiSign)

Fixes to context config merging (@zeitos)

JobSqlDAO to support username, password, DB connection pooling (@ffarozan)

Doc: Instructions for running job-server docker container in yarn-client (@MeiSign)

Doc: Various fixes to README (@noorul) and deploy instructions (@mvle @lewismc) and troubleshooting (@eranation)

Fix issue #229: Always use limit while reading from DB (@noorul)

Fix typo in SparkStreamingJob that used to be called SparkStramingJob. This will break your code if you used SparkStramingJob (@zeitos)

#Scala #akka @ApacheSpark

Spark streaming context support !! (@zeitos)

Change server_start.sh to use spark-submit. This should fix some edge case bugs.

Configurable driver memory (@acidghost)

Be able to accept environment vars in job server config files, eg master = ${?MY_SPARK_HOST} (see the Typesafe Config docs)

#Scala #akka @ApacheSpark

Updated for Spark 1.0.2