Realtime Twitter Stream Visualization with .NET Core, Emitter and JavaScript
Visualizing a stream of tweets using Emitter Publish/Subscribe in real-time. Deploying .NET Core on Linux with Docker.
Introduction
This article presents a visualization of a Twitter sample stream with a C#/.NET Core publisher application which captures the stream from Twitter, processes and re-publishes text and avatars through emitter messaging broker (also on GitHub), which allows broadcasting the stream to subscriber clients, written in Javascript.
[View Live Demo]
Background
The architecture of the application is very simple, yet powerful. We have a publisher application which consumes tweets from Twitter API and publishes it in Emitter Broker. The broker, which handles multi- and broad-casting of the messages, is used to transfer the processed tweets from the publisher to the HTML5 consumers.
Tweet Publisher (C#/.NET Core)
- Establishes connections with Twitter Streaming API and Emitter Broker.
- Every time a new tweet is received, processes the tweet and publishes it into
tweet-stream
channel.
Javascript Client
- Subscribes to
tweet-stream
channel. - Every time it receives a tweet through
tweet-stream
channel, processes it and binds it to the view.
Tweet Publisher - C#/.NET Core App
Below is most of the code I had to write for the application that consumes Twitter Stream, processes it and republishes into Emitter.
- First, it uses a very nice, and .NET Core compatible Twitter API: https://github.com/linvi/tweetinvi to hook up the Stream and a
TweetReceived
event which will be invoke every time the message is received from Twitter. - It establishes the connection with Emitter and performs Twitter Authentication. If you want to try out for yourself, make sure to get the 4 security keys required to authenticate your app, which can be found here https://apps.twitter.com.
- We use a rate limiter, allowing us to have a steady and visually pleasing rate at which we show tweets to the user.
- For each tweet received, we skip if it’s a retweet and simply re-publish the
Text
andProfileImageUrl400x400
in JSON format intotweet-stream
channel of Emitter.
public class Program
{
public static void Main(string[] args)
{
// Connect to emitter
var emitter = Connection.Establish();
// Set up your credentials (https://apps.twitter.com)
Auth.SetUserCredentials(
Environment.GetEnvironmentVariable("CONSUMER_KEY"),
Environment.GetEnvironmentVariable("CONSUMER_SECRET"),
Environment.GetEnvironmentVariable("ACCESS_TOKEN"),
Environment.GetEnvironmentVariable("ACCESS_TOKEN_SECRET")
);
// Setup a rate limiter, only re-publish one tweet per 350 milliseconds
var limiter = new Throttle(1, TimeSpan.FromMilliseconds(350));
// Using the sample stream
var stream = Stream.CreateSampleStream();
stream.AddTweetLanguageFilter(LanguageFilter.English);
stream.FilterLevel = Tweetinvi.Streaming.Parameters.StreamFilterLevel.Low;
stream.TweetReceived += (sender, t) =>
{
// Skip retweets
if (t.Tweet.IsRetweet)
return;
// Limiter allows us to have a steady, visually pleasing rate of tweets
var ct = new CancellationToken();
limiter.Enqueue(() =>
{
// Publish the tweet to the broker
emitter.Publish(
"IsQ7z18uGEFpjOJpt4K6ij49klT3PGzu",
"tweet-stream",
JsonConvert.SerializeObject(new
{
avatar = t.Tweet.CreatedBy.ProfileImageUrl400x400,
text = t.Tweet.Text
}));
}, ct);
};
// Start
stream.StartStream();
}
}
Tweet Publisher - Deploying with Docker
We are going to deploy our little .NET Core publisher using Docker. Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.
One of the easiest ways to get started is to use a Dockerfile along with Docker Hub Build Server which would build an image we could deploy on the target machine. Docker Hub allows us to automatically build a Docker Image (think of a Virtual Machine Image here) which we could deploy. The image can be triggered to automatically re-build on every commit to the git repository.
Our docker file, as shown below, does several things and essentially represents a script which is used to create the Docker Image:
- It starts with the Docker image provided for .NET Core by Microsoft.
- Updates the apt-get sources and clones the Git repository.
- Performs
dotnet restore
which essentially does anuget restore
and loads all the dependencies. - Performs
dotnet publish
which creates final binaries for our image. - Hooks a
deploy.sh
script, which will be executed every the image starts. This script simply runs our application.
FROM microsoft/dotnet:1.0.0-preview2-sdk MAINTAINER Roman Atachiants "roman@misakai.com" # Make sure we have S3 & additional libraries RUN apt-get update -qq \ && apt-get install -y git \ && mkdir /tmp/emitter \ && cd /tmp/emitter \ && git clone "https://github.com/kelindar/twitter-stream.git" "/tmp/emitter" \ && cd /tmp/emitter \ && dotnet restore -v Minimal \ && cd /tmp/emitter/src/Server \ && dotnet publish -c Release --output /app \ && rm -rf /tmp/emitter # Application will be in app folder WORKDIR /app ADD deploy.sh / CMD ["/bin/bash", "/deploy.sh"]
Once the image is built, we can start it by typing docker run -it kelindar/twitter-stream
, assuming you have docker installed. It will automatically download the image from Docker Hub and run it for us interactively.
Javascript Client
Similarly to the C# app we just wrote, the client is extremely simple and straightforward. In fact, all of the javascript (excluding dependencies) is shown below. The client does several things:
- Connects to emitter publish/subscribe service.
- Sets up our ViewModel using VueJS.
- Once we’re connected to emitter, it subscribes to
tweet-stream
channel. - Every time a message is received, we push it into the
data
property of our ViewModel, which is bound to the HTML DOM.
// Connect to emitter broker
var emitter = emitter.connect({
secure: true
});
var key = 'jtdQO-hb5jfujowvIKvSF41NeQOE8IoF';
var vue = new Vue({
el: '#app',
data: {
messages: []
}
});
emitter.on('connect', function(){
// once we're connected, subscribe to the 'tweet-stream' channel
console.log('emitter: connected');
emitter.subscribe({
key: key,
channel: "tweet-stream"
});
})
// on every message, print it out
emitter.on('message', function(msg){
// If we have already few messages, remove the oldest one (first)
if (vue.$data.messages.length >= 8){
vue.$data.messages.shift();
}
// Push the message we've received
vue.$data.messages.push(msg.asObject());
});
Most of the UI here comes from one of my previous articles where emitter was used to create a simple group chat. Feel free to check the live demo and the corresponding CodeProject Article if you’re interested.
History
- 12/04/2016 - Initial Version of the article