Tony Dark | The Black Tony Stark

So a couple of weeks ago, I posted this blog post asking for some help on redesigning Scarlett to it easier to develop her/add more features etc. Luckily the Internet gods blessed me with good fortune and I was able to receive some pretty awesome advice from a number of sources. Here I will detail all of the advice given, and some take aways I had.

Illustration by KASI MINAMI

Design pattern suggestion( Reddit ): Turn Scarlett’s Listener into a Dbus service and have her core features connect as clients.

I popped my head in over to /r/gnome, and posted this ... and /u/totallyblasted really came through for me. Besides helping me w/ my fundamental understanding of what an idle_add is, he presented a concept that blew my mind and really simplified, well, everything. See below:

/u/totallyblasted suggestion( reddit ):

Pattern seems valid on quick glance, except I’m wondering why using idle and not threads. I for one thing question something else. Wouldn’t it be better to create listener as service where clients register over dbus and where your application is the first client for it? Not only would this make reusable design for other applications as well, you’d end up with 2 simpler parts instead of one complex.

Tony Dark reply( reddit ):

Now that is a very very interesting idea /u/totallyblasted[1] re: Listener as service and the application client. Going to play with that idea on paper for a bit … Re: Threads vs idle add, I think I must just have a fundamental misunderstanding of how to call something in a separate thread that has it’s own mainloop ( ie. something using Gstreamer ). In the past, my application only survived one run before there was a stack dump, always thought it was because I was calling things improperly. I’ve definitely read https://wiki.gnome.org/Projects/PyGObject/Threading[2] several times, but perhaps I need to give it another glance over.

/u/totallyblasted suggestion (con’t):

Well, for one thing you’re not handling anything time pressing, since you’re processing voice and just send activation signals. Might be even more interesting if your service is designed to support design somewhat like this (over dbus) SomeApplication > Service - I need to know when user says “shit”, “blabla” or “somethingelse” or chains of words Service > Application - Gives session dbus back and then just signals back which registered string was said on that one whenever those occur Application > Service (later in running) - Add “bleh” to my session keywords and remove “blabla” And so on. Note, that service could serve for any app. And your application would become two things at once. Application as intended and POC for any other application interested in your service Note, that since you’re already in 2 separate applications, most of the otherwise needed threading is already gone. And the fact that you don’t process anything time pressing, but you have to process constantly without breaks makes threading more suitable.

Tony Dark thoughts:

You can follow more of the conversation over on Reddit, but what you’re really looking for is right here: https://github.com/bossjones/scarlett-dbus-poc. I created a Proof-of-concept for this new iteration of Scarlett. Yes yes, I know I could have just created another branch in the main repo, but I wanted to start fresh, get things working, then move everything back into the main repo later after I had a more complete understanding of everything. By opening multiple ssh windows and running scarlett_listener.py and scarlett_tasker.py, Scarlett comes up and starts handling requests immediately. So much cleaner, easier to make modifications to her now! This assumes you have a complete Scarlett environment put together ( Need better scripts * cough Ansible * to handle this … more later ). Here is a quick diagram on how things work using this system:

Diagram of New Scarlett Listener Dbus Service ( Showing only one signal in diagram for simplicity )

Diagram of Listener Dbus Service and Tasker connecting as client  

I also wanted to include some secondary suggestions for improving Scarlett as a result of this blog post from other developers across Reddit+ Twitter.

Suggestion: Upgrade Python 2.7 + Gstreamer 0.10

/u/Eingaica had several questions regarding the versions of Gstreamer + Python I chose to use:

I currently don’t have the time to look at the details of your system, so just a quick question: Why Python 2.7/GStreamer 0.10 and not the current versions Python 3.x/GStreamer 1.x? Also a small note: g_idle_add doesn’t use a separate thread. It just adds a function to the main loop, which will call it (from the main thread) when it’s idle.

Tony Darks Thoughts:

Moving to Python 3 isn’t something I’m completely interested in doing quite yet. Python 2.7 is going to be around for a while and has a lot of support still, so notttt yet… Moving up to more recent versions of Gstreamer, GTK, Pocketsphinx etc though … That is a must. Prerequisite work is a better way to provision a Ubuntu/Raspbian server from start to finish. Looking at Ansible to do that.

Suggestion( twitter ): Implement Scarlett w/ an Actor Model design pattern

https://twitter.com/stommepoes usually has some really awesome suggestions for Scarlett … from features ideas to application design, etc. This time around the suggestion revolved around looking into refactoring Scarlett to use a Actor Model implementation https://en.wikipedia.org/wiki/Actor_model. Currently there has not been any work on testing this implementation out since the POC worked right off of the bat … But I do plan on giving it a look at some point in the near future ( Yay Christmas break ! ) See a snippet of the convo below:

More here: https://twitter.com/bossjones/status/656195018378629120

https://twitter.com/stommepoes/status/656025216452313088

Tony Dark’s Next steps:

Now that the newly designed Scarlett POC is working, it’s time to finally port over to PyGI, Gtk3, and GStreamer 1.0. Will be using this awesome guide to do that ( https://wiki.ubuntu.com/Novacut/GStreamer1.0 ), but there are a couple prerequisites:

Need to have a better way of provisioning a vagrant box w/ all prerequisites for Scarlett from scratch. This allow others to play w/ Scarlett and contribute Pull Requests etc to the various repositories. Ansible seems like a great choice since it only requires SSH access to the machines at your disposal. This will also make configuring things like Raspberry Pi’s much easier. FYI I would like to try moving the test instance to 14.04 or even 14.10 Ubuntu Desktop ( Since Pocketsphinx master claims it supports that off of the bat ). The interesting challenge will be figuring out which parameters we need to change in the newest version of Pocketsphinx (5prealpha) to obtain the same level of accuracy that we had in version 0.8. Reason why this will be a bit of a challenge is because Pocketsphinx 5prealpha was completely rewritten in C++ and v0.8 was written in C. The values we need to alter might be in a different location/renamed/not used anymore…Won’t be the most difficult thing in the world, we always figure it out don’t we?

Research & Development on running Scarlett in a Docker container. Even easier than provisioning your own Vagrant boxes for development … using docker to simply pass around a prebuilt image will make things like running tests on TravisCI SUPER FAST. The one difficulty will be figuring out how to get a “GUI” application to work within a container … Luckily Jessie Frazelle over at Docker have an awesome blog post on running GUI apps in Docker!

(Optionally) Get everything working on Ubuntu 12.04 LTS first before switching everything else. I already have a working 12.04 Desktop instance preloaded with everything we need. But by default it is using older versions of … everything. We can still update the code to use PyGI, Gtk3, and GStreamer 1.0 but we might not see the benefits of doing that till we upgrade all other underlying libraries as well ( 12.04 LTS isn’t supported anymore )

Decisions decisions! Ill take a second to map everything out and see what to tackle first over the next couple days. If you have thoughts on what makes the most sense, drop a comment below!

‪#‎3Dprinter‬ update. My new Z coupler part finally came in the mail and I got it on correctly ( didn't use to much super glue his time!!! ). Final task... Which is a difficult one... Find a way to get alllllll of the wires through that cluttered area at the base and connected to the printer chip board. Tried taping it together and threading it through using some zip ties to pull it.... but that didn't work. Might have to try one at a time next. Tonight might finally be the night mannn.

Tony Dark made something! Yay!

So I decided to build a 3D printer this weekend... It’s basically finished but I accidentally super glued my Z coupler + screws incorrectly. Waiting on #printrbot to mail me a replacement, then will finish up and start printing things! Not 100% sure what I am going to print yet, and how it can benefit things like Scarlett, but I’m excited about having the ability to do so. Leave some ideas below of what I should print! 3d Printer: Printrbot Simple 1405.

IMPORTANT NOTE: This is a letter I’m sending to the python-hackers-list, which is comprised pygtk, pygi, and pygobject maintainers( https://mail.gnome.org/archives/python-hackers-list/ ) I wanted to cross post it just in case someone else has some suggestions for me. Let me know what you guys think in the comments below!

Hi guys,

So I have a somewhat in depth question about the design of my application. Will do my best to just highlight the important information to ensure that there is enough to describe what I’m doing and what my possible pain points might be.

Overview:

I’m working on a home automation daemon call Scarlett. Scarlett is a voice controlled python application that listens all of the time for her name, then listens for a qualifier ( Command / task to do, eg. "What's the time", "turn off lights" ) and performs an action. Once the task is done, Scarlett’s state is reset and the process begins again. Scarlett has several main features:

– Listener: Scarlett’s Listener is a subclass of GObject, and utilizes Gsteamer + Pocketsphinx to handle Speech-to-text. Pocketsphinx has been compiled against python to create a Gstreamer plugin.

– Speaker: Scarlett’s Speaker is also a subclass of GObject, and utilizes Gstreamer + Espeak via gst-plugins-espeak to handle all Text-To-Speech.

– Player: Scarlett’s sound player is also a subclass of GObject and plays static .wav files when Scarlett completes a task, or acknowledges a command, etc. Player uses Gstreamer to play sound within the application.

– Feature Objects: Scarlett Feature Objects include TimeTask, WeatherTask, TvTask, etc. Each of them is a subclass of GObject and has a number of function that allow it to obtain information or complete a given action. More on that later.

– Scarlett Daemon: Scarlett Daemon ( aka the SD ) is the glue between all of her features. It contains the GObject MainThread Loop and acts as a supervisor for all processes run in separate threads. The SD is a subclass of dbus-python module ( dbus.service.Object ), and uses gobject.GObject.connect() to connect signals with handlers within the Listener, Speaker, and Player. An example of this is:

sd.connect('rdy_for_keyword',listener_rdy)

NOTE: Where listener_rdy is defined as from scarlett.listener.listener_rdy_handler as listener_rdy

Hopefully the basics make sense, and if not, I hope they already outline any flaws.

Version information:

Since someone will probably ask this sooner than later:

Python 2.7.10

PyGst 0.10

Gobject (2, 32, 4)

What am I looking for:

I simply want advice on if I am designing my application correctly. I’ve gone through several iterations of it at this point, and this time I really want to make sure everything is architected out property before I write any more code.

Important Notes For People Not Familiar With GObject:

idle_add is a GObject function that lets you run something in a separate thread.

emit sends a signal to another object and does something based on that.

In the diagrams below, anywhere that says 'IA:' means idle_add.

In the diagrams below, anywhere that says 'E:' means emit.

Diagram:

The following 2 pictures show my proposed design for my application.

Without Numbers:

With Numbers:

It is meant to describe from start to finish what it is like issue a command, start a task, then finish. Blue lines represent a GObject.idle_add, and Green lines represent a GObject.emit. Anything that has 2 direction arrows means a signal is emitted when a task is finished to Scarlett Daemon, which is where we run our mainloop / supervisor.

Walking through the process:

=================

Scarlett is started, Gobject’s Mainloop is in Scarlett daemon. If you’re curious, here’s the code that handles that here: https://gist.github.com/bossjones/7ce13ade25941b34b63f

Using a gobject.idle_add, Scarlett Daemon starts the Listener in a separate thread.

Using a gobject.idle_add, Scarlett Daemon starts the Player in a separate thread.

Using a gobject.idle_add, Scarlett Daemon starts the Speaker in a separate thread.

After all threads have been started, Scarlet Daemon sends a signal to the Listener via a gobject.GObject.emit saying 'listener_rdy' to let it know that it should start running Gstreamer which will bring up a long-running Speech-To-Text process.

Scarlett Daemon does a gobject.GObject.emit() saying 'play_sound_ready' to the Player thread to announce that everything is up and running. This starts a short lived Gstreamer run that simply plays a beep sound to acknowledge that the task happened. The Player thread then emits a signal back to the Scarlett Daemon telling it 'play_sound_fin'.

Scarlett Daemon emits signal to Listener thread 'rdy_for_keyword', this alerts the Listener to begin listening specifically for keyword: 'Scarlett'.

After 4 failed attempts Listener emits 'max_listener_fails_reached' to Scarlett Daemon to restart the entire process. Failed attempts are stored in a GObject property named 'stt_failures' ( default value: 0 ). If it find a correct match, it emits 'keyword_found' to the Scarlett Daemon instead and sets GObject property 'stt_failures' back to its default 0. Listener sets the GObject property 'kw_found' to 1 to keep the state of the object.

Scarlett Daemon emits 'play_keyword_found' to the Player thread which kicks off a gstreamer run that simply plays a beep sound to acknowledge that the task happened. When finished playing, Player emits 'keyword_found_fin' to Scarlett Daemon.

Scarlett Daemon, which already emitted 'keyword_found' to the Listener, now emits 'rdy_for_cmd' to the Listener thread to let it know we are ready to find a qualifying command. eg 'what time is it', 'turn on the lights', 'turn on the tv', etc.

After 4 failed attempts Listener emits 'max_listener_fails_reached' to Scarlett Daemon to restart the entire process. Failed attempts are stored in GObject property 'stt_failures'. If it find a correct match, it emits 'cmd_found' instead and sets GObject property 'stt_failures' back to its default ‘0’.

Scarlett Daemon receives 'cmd_found' signal and brings up a feature object in a new thread using an gobject.idle_add. These objects, like everything else in this project is a subclass of GObject ( ex. TimeTask, WeatherTask, TvTask, etc ).

The correct command task is run, and emits 'task_action_fin' to Scarlett Daemon when finished. ( eg 'tv_poweron_fin' )

Scarlett Daemon emits 'scarlett_speak' to Speaker thread which kicks off a Gstreamer run using gst-plugins-espeak to perform a Text-To-Speech.

When Speaker is finished running, it emits 'speaker_fin' to Scarlett Daemon.

Scarlett Daemon emits 'play_sound_complete' to Player thread which uses Gstreamer to play a wav file as acknowledgement of job being finished. It emits 'play_sound_fin' when done to Scarlett Daemon

Scarlett Daemon emits 'scarlett_run_fin' to Listener which tells it to reset all GObject properties back to their default.

Listener emits 'listener_reset_fin' to Scarlett Daemon when finished.

Loop and repeat!

Sorry if any of that was dragged on too long, but wanted to make sure I wrote that out as clear as possible.

My questions:

=================

Does this sound like a valid design pattern for my application and will it work with the python GObject bindings? I understand I might need to add a couple threading objects to prevent things from running at the same time ( threading events, locks, rlocks, etc )

Does the long running Listener thread which does Speech-To-Text in the background make sense and will it work as intended?

Thanks for taking the time to read this friends, hope to hear from you soon!

Hey friends! Back with a quick blog update for ya. For several years now i've been a big fan of Evernote and their Web Clipper ability. Super useful especially combined with a premium account and an iPad mini ( offline mode FTW! ). But I soon ran into an issue when I noticed this during my python/gstreamer research:

ZOMG, 314 ITEMS?!?!!?

I use that web clipper like a mad man!!! Sometimes I just clip things just to clip it! That made it super difficult to figure out where I left off in my R&D for Scarlett and what articles I should be reading next to help further my journey.

While re-watching this scene from Iron Man, it struck me, I needed a journal to keep status updates of important discoveries throughout this process:

Yeah yeah yeah, I know Tony Stark is merely talking to Jarvis and Jarvis is acting as his personal journal, but Scarlett doesn't quite have that feature yet :)

Till then, I've decided to create a separate twitter account dedicated strictly to Scarlett development. Introducing @scarlettapp

Tweets by @scarlettapp // <![CDATA[ !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs"); // ]]>

This will help as I continue to iterate on Scarlett and discover new strengths/weaknesses in her code base!This will especially become useful in the upcoming major change involving moving from Gstreamer 0.10 -> 1.0 and GTK+2 -> PyGI/GTK+3.

Make sure you tune in there if you want to know what's new! NOTE: Don't worry I'll still be tweeting from @bossjones as well.

Look out for another blog post this week on setting up a python virtualenv with PyGI/GTK+3/Gstreamer 1.0 !

Till next time.....

-- Tony Dark ( and #gastonthefrenchie, new assistant )

PS Check out these cool videos too:

While doing some research on gevent to make sure I know about all of the "gotchas", I came across this answer on Stackoverflow. Super helpful.

So I was shooting the shit with my buddy @jackyalcine the other day, and he brought my attention to a couple of other very cool home automation projects that are out there, eg api.ai and wit.ai. I also decided to open up hubot on github just for the hell of it to see how they were dealing with sharing resources, states, and events throughout objects. I realized several things:

1. I noticed that a lot of these projects are implementing their own DSLs. Whats a DSL? According to this [stackoverflow post]:

Domain-specific language (DSL) is a programming language or specification language dedicated to a particular problem domain, a particular problem representation technique, and/or a particular solution technique.

2. I also noticed that these same applications are using a PubSub design when implementing their applications to alert different objects, services, etc of changes to the state of the application, eg "Are the lights on or off right now", "When was the last time pocketsphinx failed to recognize a keyword", etc etc.

I started to realize that Scarlett is lacking in those departments. Currently it is not easy to define a new service that Scarlett can integrate with. Now... going the route of a DSL might be a little overkill for what we're trying to do right now, there are some steps can be made to define what a very basic service in Scarlett should be able to do( Especially since things are must more object oriented now, yay inheritance ). So i'm left to figure out one more thing ... How do you pass around information regarding the state of Scarlett and everything she interacts with? Whenever she's rebooted, she has no idea if the lights are on/off, if she should be processing a task in the background or anything! What should we do?

Enter the Publish-subscribe Pattern.

What does it mean?

According to wikipedia

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called > publishers, do not program the messages to be sent directly to specific receivers, called subscribers. > Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers > there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that > are of interest, without knowledge of what, if any, publishers there are.

Pub/sub is a sibling of the message queue paradigm, and is typically one part of a larger message-oriented > middleware system. Most messaging systems support both the pub/sub and message queue models in their API, e.g.> Java Message Service (JMS).

This pattern provides greater network scalability and a more dynamic network topology, with a resulting > decreased flexibility to modify the Publisher and its structure of the data published.

I will now begin working on creating an Event Emitter for Scarlett. Each object will be able to subscribe to a Redis backed channel and will be able to publish their current state, subscribe and see if there was a change in their state, if a keyword was identified by [pocketspinx], any type of failures that occurs, etc. All of Scarlett's services( brain,listener,hue lights,window blinds,sonos,forecast) will now be fully aware of what the other is doing. This will also be helpful as we move towards the greater goal of splitting up different tasks onto different nodes. Since we're using Redis, Scarlett can now pick up right were she left off after a system reboot, yay for persistence! Lucky for us, the amazing redis-py module has an awesome PubSub class which will allow us to do this all very easily.

While thinking about this new design approach, I automatically thought about a potential bottleneck we might encounter.... Listening on the redis channel is a blocking operation, meaning, by default only one object can access it at a time, and that'll prevent the rest of the application from running. How do we solve that?

gevent to the rescue !

What is gvent?

gevent is a coroutine-based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev event loop.

What else can it do?

Fast event loop based on libev (epoll on Linux, kqueue on FreeBSD).

Lightweight execution units based on greenlet.

API that re-uses concepts from the Python standard library (for example there are Events and Queues).

Cooperative sockets with SSL support »

DNS queries performed through threadpool or c-ares.

Monkey patching utility to get 3rd party modules to become cooperative »

Basically, by using gevent, we'll be able to achieve long polling for Scarlett, and allow her to periodically query for changes in the state of anything she's connected to. AWESOME!

balloob over at github has an awesome home automation project where he identified a very smart and straight forward architecture diagram for his project, check out this image below:

We will be implementing something very similar for Scarlett. Pretty soon we'll be able to move into multiprocessing in Python, boy oh boyyyyy! Get ready for the next wave of awesomeness!

I'm going to be using the following articles as resources to implementing the ScarlettEventEmitter:

Gevent, Long-Polling & You

Concurrent connections to Redis with gevent and redis-py

As always, you can keep up with what's going on by checking out the Scarlett repo on github!

Friends! I'm back !!!!

As some of you know, the past couple months have been pretty crazy between stuff at the office & working on getting a new apartment etc. Well guess what, life has finally chilled out a bit and I can rededicate some time to my favorittttte girl, Scarlett.

What's different this time around?

1. I bought an apartment in NYC! Why the hell does this matter? Well since I'm a homeowner now, I can rip up any floor I want, change any fixtures I want, etc etc. We now have the creative freedom to do whatever we want with Scarlett. Nice!

2. I've learned A LOT about Python over the past couple months from working on other projects at behance/on-the-side. This means debugging Scarlett code should be much faster now ( sheesh how did I do anything without virtualenv, ipython and auto-complete ), and i'm adding something great ... TESTS! Yes, all Scarlett code will now be tested in travis.ci. Ill start off with unit tests, then later ill implement some integration tests, and a couple more. Going to take a little time to get everything green, but this will ensure that I don't keep breaking stuff by switching code around all over the place. I've completely refactored the original repo to make it look a lot more like a python application. 

3. I'm segmenting things out more & using the right languages to get every job done. The feature list for Scarlett has gone through the ROOF. So many things I want to add all over the place. This new version of Scarlett will introduce a Plugin/Features class which will allow you to chose which features you want to active via your config file ( forecastio, spotify, hue lights, etc etc ). This means you wont have to use features if you don't want to, simple configuration for the win! Also, you'll be able to extend and create your own Plugins! I'm also going to be embracing C a lot more for certain tasks...speed speed speed. Couple ideas for things like zeromq, flask etc that I have... but I'll save that for a future post. 

4. Loose coupling to save the day. Each "part" of Scarlett will be made into a separate service. I like the idea of having several scripts doing one task VERY WELL, instead of having one gigantic script. Easier to debug + optimize if we take this approach as well. Pick this up at work, and its been helping us out with handling some very BIG projects across the board.

5. Documentation. Im writing down every damn thing. Good practice and easier for others to contribute.

6. Exploring multiple HW devices. I'm still working with the Raspberry Pi, but also experimenting with the Beaglebone, an old Thinkcentre, and my Virtualbox dev machine. I'm doing this to compare performance metrics based on system specifications etc. Even playing around with the concept of creating a Beowulf cluster. More updates coming, get ready for awesomeness!

NEW TOYS. Got me a HVAC IR REMOTE MODULE FOR ARDUINO / RASPBERRY PI. Can't wait to play with DAT.

NOTE: THIS BLOG POST IS A COUPLE WEEKS OLD BUT I DIDNT TAKE THE TIME TO READ THROUGH ALL OF IT AND POST IT IN A TIMELY MANNER. ANOTHER BLOG POST COMING SOON! What's up family, been a while, but wanted to let you guys know I'm still actively working on everything. Couple important updates I wanted to make you aware of.

My biggest goal at the moment is getting my Arduino Uno to control my Apple TV (x2), Toshiba 42" LED TV, and my AC Unit. Once I can do this regularly from some testing platform, I'll integrate it into Scarlett's voice recognition system on my Raspberry Pi.

SPOILER ALERT ... I've succeeded!! But figured out a couple things I need to fix/tweak/recognize to make this system a little more robust. Lets start at the beginning of this feature.

NOTE: For the testing of this feature, I've been using my [ Macbook Pro ] to develop on, simply because I can move around from room to room and test, etc.

How did I approach this?

Day in and day out at Behance, I work in a terminal window trying to connect clusters of servers together, in an effort to make them talk to each other, run tasks, etc etc. Most of the time, I accomplish this by using some sort of Command Line Interface tool ... so it seemed like the appropriate approach to take w/ this TV feature. I wrote a quick CLI tool called ScarlettTV CLI in Python . ScarlettTV CLI connects to your Arduino via a serial port ( I used the pyserial library btw ) and runs arbritary code on your Arduino.

Research

There's a million different blog posts out there on teaching your Arduino how to decode/send IR signals to something ( Special thanks to my brother Joshua Arenas for the crash course on how to use an [ Arduino Uno ]! ).

My favorite ones are by Ken Shirriff , who created the awesome Arduino-IRremote library :

Blog post - Controlling your stereo over the web with the Arduino infrared library

And Cyborg5 , who has refactored Ken Shirrif's code and extended it with IRLib :

Blog post 3 out of 3 - IRLib Tutorial part 3a: Sending IR Codes

Read any of their blog posts to figure out how I was able to decode all of the remotes in my house. They're very well written and extremely descriptive. I will post another blog post later on with details on how I setup my Arduino to send IR signals complete with an up to schematics, diagrams, etc once i finalize its design, currently I don't like how many wires I have lying around, want to compact things a bit and make it sleeker & a bit more portable.

Btw, if you're thinking about doing write ups about your Arduino project, make sure you check out Fritzing ( [ ITS SUPER WAVYYYY ], B ).

According to their website:

Fritzing is an open-source hardware initiative that makes electronics accessible as a creative material for anyone. We offer a software tool, a community website and services in the spirit of Processing and Arduino, fostering a creative ecosystem that allows users to document their prototypes, share them with others, teach electronics in a classroom, and layout and manufacture professional pcbs.

Example diagram:

http://fritzing.org/projects/arduino-presenter-receiver

Playing with ScarlettTV CLI

As I mentioned, I've been doing a lot of this project in Python ... please bare with me cause I'm still learning how to do things the "Python way". Eventually I'll sit down and go through the correct guides when I refactor everything, but for right now my focus is results, not pretty code. :) Feedback is certainly WELCOME though, tell me how to make things CLEANER.

NOTE: From what I understand, you should ALWAYS develop in Python using Virtualenv to keep your projects and their dependencies separate. A perfect guide on doing this can been seen here via the guys at [ Digital Ocean ]: https://www.digitalocean.com/community/articles/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

Using pip I installed the following on my Macbook Pro ( NOTE: Not my virtualbox setup yet ) before starting anything:

pip install pyserial==2.7 pip install argparse==1.2.1

Going to need both of these in order to make the python script run correctly.

Running it with the --help flag will tell you the basics.

malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts: ./scarletttv_cli --help usage: CLI tool to control Arduino IR signals. [-h] [--doctest] [-v] [-V] [-d {frizzytv,appletv,toshiba}] [-c COMMAND] [-t] [-p PORT] optional arguments: -h, --help show this help message and exit --doctest perform doc tests -v, --verbose be verbose in all things, go with god -V, --version show version -d {frizzytv,appletv,toshiba}, --device {frizzytv,appletv,toshiba} set device to one of several ( eg. frizzyTV ) -c COMMAND, --command COMMAND Set command -t, --test Run serial port test suite -p PORT, --port PORT Set the serial port malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts:

If you run it with the --test flag, the cli tool will print a list of available ports.

malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts: ./scarletttv_cli -t /dev/cu.Bluetooth-Modem /dev/cu.Bluetooth-PDA-Sync /dev/cu.Bluetooth-Serial-1 /dev/cu.Bluetooth-Serial-2 4 ports found malcolm@malcolm-MacBookPro1 [feature-parse-launch*] ~/dev/scarlettPi/scripts:

After identifying the IR codes for all of my remotes, I made use of Python's powerful dictonary data structure ( In other languages, this is simply an associate array ). I stored all of my device names, their commands, and the proper IR Hex code associated with sending that command. Lets take a look at the Apple TV section in depth a bit more.

command_master_list = { "appletv": { "up": "1,77E1D030,32", "down": "1,77E1B030,32", "left": "1,77E11030,32", "right": "1,77E1E030,32", "menu": "1,77E14030,32", "pause": "1,77E17A30,32", "play": "1,77E17A30,32", "circle button": "1,77E1BA30,32" },

So again, device name we're interested in is appletv.

If you look at an Apple TV remote ( 3rd Generation ), they're designed to be very simply like most of their products. They only have a couple of different button options: up,down,left,right,menu,pause/play,circle button/enter. That's it! I decoded all of their hex codes and place it into a string. To read the command format, the first number indicates the protocol number of the IR signal, which is NEC in this example. The second number is the hex code to be transmitted, and the final number is the number of bits ( 32 bits for all Apple TV commands ).

To see more examples of this, please check out Cyborg5's IRserial_remote.py example on github.

IMPORTANT ASSUMPTION: Right now we're assuming your Arduino has already been pre programmed to send IR signals. I modified Cybog5's IRserial_remote.ino sketch to work for my needs. Again, we will go over this in more depth in another blog post focused more on the Arduino.

Now that we have those defined, I simply run with the following flags while my Arduino is connected to my MBP and pointed directly at my Apple TV:

./scarletttv_cli --device appletv --command left

And boom, I'm now controlling my Apple TV from the command line!

Hey guys,

Super sorry I've been M.I.A guys. Two of my buddies ( #Dwalk + #Savage ) mentioned to me the other day that I made so much of a big deal about everything involving this project... then just dropped it all....Things got really busy and I had to refocus on a couple of things, but I'm back and in full force! Over the past two days, i've been working on Scarlett again, and i've made some huge progress.

NOTE: When you working on something big, but you come to a wall...step back...look at old code with new eyes, and a new approach, and you'll come up with something.

The Black Tony Stark is back:

---

New Features

API integrations? Voice controlled Hue lights? You asked for it, I did it. Here's a couple videos showing it in action:

Pocketsphinx + Philips and Hue Lights Integration:

---

Lets try changing the colors of the lights now .....

---

Gave her a voice using Festival ( Chose this over eSpeak ):

---

Integration with forecast.io, now I know the weather every morning:

Do you guys have any ideas for me?

Any feature integration ideas?

Please let me know, I want more idea!

So a lot of people have been asking me, what do I plan on using to make this whole thing happen? Well, lets go over it:

Hardware Requirements:

  Raspberry Pi

NOTE: I HIGHLY suggest getting this, it’ll make sure your steps are EXACTLY the same as mine:

CanaKit Raspberry Pi Complete Starter Kit Which includes a Raspberry Pi 512 MB + Black Case + Micro USB Power Supply + Original Preloaded SD Card + HDMI Cable

Playstation 3 Eye

HDMI Cable

USB Wifi - Edimax EW-7811Un 150 Mbps Wireless 11n Nano Size USB Adapter with EZmax Setup Wizard

Flash Memory Card - Transcend 8 GB Class 10 SDHC (TS8GSDHC10E)

Ethernet cable

USB keyboard

Hardware Suggestions ( If you want to do cooler stuff! )

Airplay capable speakers ( Something to allow your J.A.R.V.I.S system to speak back to you ! )

I used this specifically:

iHOME iW2 AIRPLAY

Another suggestion: Apple TV

Hue Philips 431643 Personal Wireless Lighting, Starter Pack, Frustration Free ( Control your lights wirelessly ! )

Bluetooth + Wifi USB - PLANEX 2 in 1 Micro Size USB Bluetooth 3.0 WiFi Combo Adapter compatible Bluetooth Ver.3.0 ( Raspberry Pi only has 2 USB ports, why not combine wifi + bluetooth ? )

Apple Wireless Keyboard

Apple Magic Mouse

Arduino

I used an Alamode specifically ( Integrated directly with a Raspberry Pi, CLEAN! )

  Software Requirements ( Don’t worry, I’ll show you how to install and configure these below in another post ):

Raspbian OS - OS for your Pi

Git - best choice for version control and collaborating

Gstreamer 0.10 ( linux ) + All plugins. ( Used to stream audio over network - Details below… )

Xcode + Homebrew - need this to install linux packages on OS X

Pulseaudio - linux audion server for input/output

Python 2.7 + couple plugins ( Details below )

Motion - will be used for motion detection

Imagemagick - image manipulation on linux

sox - sound swiss army knife

mplayer - media player on linux

ffmpeg - Cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Airplay / Shareport service ( linux ) - allow raspberry pi to stream audio via airplay

arecord - record audio on linux

pocketsphinx - speech recognition tool on linux

eSpeak - text to speech on linux

Festival - Festival is a general purpose text-to-speech system. As well as simply rendering text as speech it can be used in an interactive command mode for testing and developing various aspects of speech synthesis technology.

nodejs 0.8.x series - Event-driven I/O server-side JavaScript environment based on V8.

If This Then That -Integrates and Automates Your Online Tools

  Software Suggestions:

Hubot- awesome little robot service that exists to do tasks for you. can talk to anything that has an API

Optional hubot-imessage - Message your Hubot system from your iPhone, ANYWHERE.

hue-cli - control your hue lights from command line!

Future Software additions ( Won’t be able to get to this by my end of summer deadline )

node-sentiment ( Give ScarlettPi some personality by allowing her to analyze sentiments of what you say )

Animate ScarlettPi’s face in bash using this:

http://evilzone.org/creative-arts/nyan-cat-ascii/

http://stackoverflow.com/questions/238073/how-to-add-a-progress-bar-to-a-shell-script

Dashing - amazing customizable dashboard that you can use to display whatever infomation you want in a beautiful interface. ties into any API.

Gearman - Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.

  API Keys Required ( If you want my entire setup )

Wordnik - online word lookup

Wolframalpha - think ask jeeves from the 90’s on steroids. i think siri actually uses this to givd you answers to… anything

Forecast.io - get weather based on location

Foursquare - location recomendations / locate friends nearby

Twitter - tweet !

Spotify - best music service out

API’s for the future ( Won’t be able to get to this by end of my summer deadline )

Yahoo Finance - stock information. gotta keep checking on that Adobe Stock!

Behance - duhhhhh, wi work there

This list will continue to grow and change. PLEASE SUGGEST MORE! And if you’re unsure of how I’m going to use some of these, please don’t hesitate to ask.

Tools I’m using to do all of this

ssh - im a terminal screen type of guy. no GUI’s for me!

vim - the Best text editor on linux. take that nano and emacs

textmate - my text editor of choice on os x

filezilla - file syncing local dev via SFTP to raspberry pi

Thats all for now! Please excuse any typos or errors. difficult to write this thing up in HTML using just my iPad. Will give it a once over again when I have my computer again. Leave questions / comments below!

"So truthfully, I wanted to wait till i'm done with the whole project to post something like this, but this was such a pain in my ass, I wanted to post my work in case anyone else is dealing with the same thing as we speak. Hope this helps someone with their work." -- bossjones

Took me a while, with with some help from a couple sources ( they will be listed in my answer ) and some helpful hints from [nikolay-shmyrev](http://stackoverflow.com/users/432021/nikolay-shmyrev), I finally came up with an answer that worked for me. Key assumptions: 1. running these commands as the pi user ( previously I was running them as root, which was incorrect ) 2. I'm using continuous recognition and I was ONLY looking for the ability to "wake-up" my raspberry pi. Upon waking it up, I have other plans on how it should interact. My setup: 1. Canakit RaspberryPi 2. HDMI cable to my Toshiba TV 3. usb wifi dongle 4. Playstation 3 Eye for speech recognition Moving forward. I ran the following commands on my RaspberryPi to get PulseAudio + pocketsphinx working together w/ my Playstation 3 Eye. ( If you see any places for improvement please let me know ) Install pulse audio / development packages

sudo apt-get install gstreamer0.10-pulseaudio libao4 libasound2-plugins libgconfmm-2.6-1c2 libglademm-2.4-1c2a libpulse-dev libpulse-mainloop-glib0 libpulse-mainloop-glib0-dbg libpulse0 libpulse0-dbg libsox-fmt-pulse paman paprefs pavucontrol pavumeter pulseaudio pulseaudio-dbg pulseaudio-esound-compat pulseaudio-esound-compat-dbg pulseaudio-module-bluetooth pulseaudio-module-gconf pulseaudio-module-jack pulseaudio-module-lirc pulseaudio-module-lirc-dbg pulseaudio-module-x11 pulseaudio-module-zeroconf pulseaudio-module-zeroconf-dbg pulseaudio-utils oss-compat -y

Setting up ALSA Per instructions from http://forums.debian.net/viewtopic.php?f=16&t=12497

sudo \cp -pf /etc/asound.conf /etc/asound.conf.ORIG sudo echo 'pcm.pulse { type pulse } ctl.pulse { type pulse } pcm.!default { type pulse } ctl.!default { type pulse }' > /etc/asound.conf

Make sure your camera device loads on boot

_DEVICE_LOAD_ON_START=$(grep "snd.bcm2835" /etc/modules | wc -l) if [[ "${_DEVICE_LOAD_ON_START}" = "0" ]]; then sudo \cp -pf /etc/modules /etc/modules.ORIG sudo echo "snd-bcm2835" >> /etc/modules fi # Disallow module loading after startup. This is a security feature since it disallows additional module loading during runtime and on user request. _DISALLOW_MODULE_LOADING=$(grep "DISALLOW_MODULE_LOADING=1" /etc/default/pulseaudio | wc -l) if [[ "${_DISALLOW_MODULE_LOADING}" = "0" ]]; then sudo \cp -pf /etc/default/pulseaudio /etc/default/pulseaudio.ORIG sudo sed -i "s,DISALLOW_MODULE_LOADING=1,DISALLOW_MODULE_LOADING=0,g" /etc/default/pulseaudio fi

Set up the PulseAudio daemon for network connections # allow other clients on the network to connect to pulseaudio daemon ( only add auth-anonymous=1 if you know EVERY machine on your LAN ... this could be a security risk otherwise )

sudo \cp -fvp /etc/pulse/system.pa /etc/pulse/system.pa.ORIG sudo echo " # ScarlettPi ADDED THIS load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;192.168.0.0/24 auth-anonymous=1 load-module module-zeroconf-publish" >> /etc/pulse/system.pa sudo echo " # ScarlettPi added this #load-module module-native-protocol-tcp #load-module module-zeroconf-publish load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;192.168.0.0/24 auth-anonymous=1 load-module module-zeroconf-publish" >> /etc/pulse/default.pa # check to make sure it looks okay cat /etc/pulse/default.pa

Change default sound driver from alsa to pulseaudio

sudo \cp -fvp /etc/libao.conf /etc/libao.conf.ORIG sudo sed -i "s,default_driver=alsa,default_driver=pulse,g" /etc/libao.conf # daemon settings according to Pi-Musicbox ( https://github.com/woutervanwijk/Pi-MusicBox ) sudo \cp -fvp /etc/pulse/daemon.conf /etc/pulse/daemon.conf.ORIG sudo echo " # ScarlettPi added this high-priority = yes nice-level = 5 exit-idle-time = -1 resample-method = src-sinc-medium-quality default-sample-format = s16le default-sample-rate = 48000 default-sample-channels = 2" >> /etc/pulse/daemon.conf.ORIG

Add `pi` user to the pulse access group

sudo adduser pi pulse-access # shut down the machine to make sure all the settings we just made are loaded correctly sudo shutdown -r now

Make sure to add `/usr/local/lib` to library path

export LD_LIBRARY_PATH=/usr/local/lib export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # also add these to your .bashrc so they get set once you login echo " # scarlettPi added this export LD_LIBRARY_PATH=/usr/local/lib export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig" >> ~/.bashrc

Install base PocketSphinx

# install python dev packages sudo apt-get install python2.7-dev -y # sphinxbase install ( required to install pocketsphinx ) sudo apt-get install bison -y cd ~pi/ wget http://downloads.sourceforge.net/project/cmusphinx/sphinxbase/0.8/sphinxbase-0.8.tar.gz tar -xvf sphinxbase-0.8.tar.gz cd sphinxbase-0.8 ./configure make sudo make install cd - # pocketsphinx install # set this: LD_LIBRARY_PATH=/path/to/pocketsphinxlibs /usr/local/bin/pocketsphinx_continuous # http://www.voxforge.org/home/forums/message-boards/speech-recognition-engines/howto-use-pocketsphinx wget http://sourceforge.net/projects/cmusphinx/files/pocketsphinx/0.8/pocketsphinx-0.8.tar.gz tar -xvf pocketsphinx-0.8.tar.gz cd pocketsphinx-0.8 ./configure make sudo make install cd - # install sphinxtrain wget http://sourceforge.net/projects/cmusphinx/files/sphinxtrain/1.0.8/sphinxtrain-1.0.8.tar.gz tar -xvf sphinxtrain-1.0.8 cd sphinxtrain-1.0.8 ./configure make sudo make install cd -

Check if pulse daemon is running

ps aux | grep pulse

# If it isn't, start it up yourself ( need to figure out the best way to make this run on boot...init.d script maybe? )

/usr/bin/pulseaudio --start --log-target=syslog --system=false

Finally, run Sphinx **IMPORTANT NOTE** YOU HAVE TO BE USER PI AND THE PULSEAUDIO SERVER NEEDS TO BE RUNNING Assumimg existing corpus file, .jsgf file, .dic, and .lm files (using [lmtool](http://www.speech.cs.cmu.edu/tools/lmtool.html))

cd ~pi/pocketsphinx-0.8 pocketsphinx_continuous -lm /home/pi/scarlettPi/config/speech/lm/scarlett.lm -dict /home/pi/scarlettPi/config/speech/dict/scarlett.dic -hmm /home/pi/scarlettPi/config/speech/model/hmm/en_US/hub4wsj_sc_8k -silprob 0.1 -wip 1e-4 -bestpath 0

References: * [Advice on how to calibrate pocketsphinx correctly](http://sourceforge.net/p/cmusphinx/discussion/help/thread/49e34dff/) * [How to get pocketsphinx to recognize new words via a corpus](http://www.ghatage.com/2012/12/make-pocketsphinx-recognize-new-words/) * [BEST/Simplest explanation of how Java Speech Grammar Format works](http://puneetk.com/basics-of-java-speech-grammar-format-jsgf) I plan on adding more details behind why I used certain setting, configurations in a blog post i'm writing on my home automation project, but figured, i'd share what i've done thus far incase someone else was stuck like me and would like to move forward with what they're working on. Hope this helps someone. Thanks for the advice guys.