Does node.js not include packages in opkg yet?

Steve Fister

@Boken-Lin Ok, so I'm having an issue running a node package on the Omega. Node works fine for simple things, but with the node-tweet-stream NPM package (installed on my Mac first, then copied over to the Omega), I continue to receive an error (see below). I backed my Mac node version down to v0.10.5 and ran the script without error. However, when the same exact script is run on the Omega, I get the following error. Note that node-tweet-stream is an uncompiled module....any ideas??

/mnt/sda1/node-tweet-stream/node_modules/node-tweet-stream/lib/twitter.js:270
  this.stream.abort()
              ^
TypeError: Cannot call method 'abort' of null
    at Twitter.abort (/mnt/sda1/node-tweet-stream/node_modules/node-tweet-stream/lib/twitter.js:270:15)
    at null.<anonymous> (/mnt/sda1/node-tweet-stream/node_modules/node-tweet-stream/lib/twitter.js:257:10)
    at Request.EventEmitter.emit (events.js:95:17)
    at IncomingMessage.<anonymous> (/mnt/sda1/node-tweet-stream/node_modules/node-tweet-stream/node_modules/request/request.js:965:12)
    at IncomingMessage.EventEmitter.emit (events.js:95:17)
    at readableAddChunk (_stream_readable.js:132:14)
    at IncomingMessage.Readable.push (_stream_readable.js:113:10)
    at HTTPParser.parserOnBody [as onBody] (http.js:140:22)
    at CleartextStream.socketOnData [as ondata] (http.js:1535:20)
    at CleartextStream.read [as _read] (tls.js:470:10)

Boken Lin

@Steve-Fister Hmmm, that's very strange. From the error message it seems that this wasn't correctly referenced, so it's an empty object. What was the line in your code that triggered this error?

Steve Fister

@Boken-Lin Well, it appears that something is missing from either the node.js package or from the OS. I've tried several twitter node modules, all with varying degrees of failure (yet work on my Mac perfectly fine). In particular, the errors seem to indicate the problem is with authenticating to Twitter apis (I have verified the credentials do work as this EXACT same script runs on the Mac).

My code running in index.js is very simple:

var Twit = require('twit')

var T = new Twit({
    consumer_key:         'XXX'
  , consumer_secret:      'XXX'
  , access_token:         'XXX'
  , access_token_secret:  'XXX'
})

//
//  filter the twitter public stream by the word 'mango'.
//
var stream = T.stream('statuses/filter', { track: 'mango' })

stream.on('tweet', function (tweet) {
  console.log(tweet)
})

Steve Fister

Oh, and this was a different node package called "twit". Here is the error received from running this:

{ [Error: Bad Twitter streaming request: 401]
  message: 'Bad Twitter streaming request: 401',
  code: null,
  allErrors: [],
  twitterReply: '<html>\\n<head>\\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\\n<title>Error 401 Unauthorized</title>\n</head>\n<body>\n<h2>HTTP ERROR: 401</h2>\n<p>Problem accessing \'/1.1/statuses/filter.json?track=mango\'. Reason:\n<pre>    Unauthorized</pre>\n</body>\n</html>\n',
  statusCode: 401 }

Boken Lin

@Steve-Fister This is helpful. I'm guess the error encountered with the other twitter plugin is the same thing. this referred to the request object, and because twitter responded with an error, it didn't contain a stream object. Now let's try to figure out why twitter responds with 401. Twitter says that 401 message means the following:

Authentication credentials were missing or incorrect.
Also returned in other circumstances, for example all calls to API v1 endpoints now return 401 (use API v1.1 instead).

But you are saying that you are using the same code on both instances?

Steve Fister

@Boken-Lin Well I think the "this" problem was specific to that script because it couldn't authenticate and as such, did not receive a stream. I've tested three different Twitter packages and they all pretty much fail with an authentication error. And, like I also mentioned, if I run the exact same scripts on my Mac, they all run just fine. These scripts don't all use the OAuth libraries either, so it's not and OAuth issues either. I'm thinking either the port of node v0.10.0 on OpenWRT is not complete, or there is a fundamental problem at the OS level on the Onion. However, I think it's more at the Node level, because I can run a curl script against the Twitter streaming API and authenticate just fine.

Boken Lin

@Steve-Fister Can you send me a copy of your sources? I would like to try it out to figure out what the issue is. I've been able to make HTTP requests using node without much issue on the Omega, and I don't think the twitter login system requires any special HTTP request features.

Steve Fister

@Boken-Lin You can grab one from here that I am currently using: https://github.com/ttezel/twit

I've also tried:

https://github.com/SpiderStrategies/node-tweet-stream
https://github.com/desmondmorris/node-twitter

FWIW, I don't think its a HTTP request problem either. I'm hoping you find similar issues (although I would LOVE to get this working!), maybe it will identify an issue somewhere.

Thanks!

Boken Lin

@Steve-Fister Ok. I'll try it out and let you know!

Steve Fister

@Boken-Lin FWIW, I have tested the node.js request module and have verified that I can indeed receive content from a HTTP request.

Boken Lin

@Steve-Fister Yeah. I'm still trying to pinpoint the issue that's causing the error. So far all HTTP requests have been okay.

Chris Ward

@Steve-Fister @Boken-Lin I have given this a try, maybe I have a bit more info.

First I copied over twit. I built it on my mac. I ran into issues with 2 things, har-validator and uuid. These both have bin files. The error I get actually has to do with creating the symlinks when extracting the tar file.

tar: can't create symlink from ./twitter/node_modules/twit/node_modules/request/node_modules/.bin/har-validator to ../har-validator/bin/har-validator: Operation not permitted
./twitter/node_modules/twit/node_modules/request/node_modules/.bin/uuid
tar: can't create symlink from ./twitter/node_modules/twit/node_modules/request/node_modules/.bin/uuid to ../node-uuid/bin/uuid: Operation not permitted

I tried to create these myself but wasn't able to. I then ran the simple hello world example, which doesn't use streams like Steve was using, and I don't get a code error, but it comes back telling me it can't authenticate. I tried this same example on my mac it is is successful.

Twit has one dependency, request, which has many including the two mentioned above. I decided to create another project using just it - no twit. I get the same errors about the bin files, but I was able to connect to and open and display a web page. So the basic request functionality is working, but not when it comes to authenticating. I can see that there would be issues with the bin files if they were used. Not sure what har-validator is needed for.

I guess we could try to cross-compile request to get the right form of the binaries and try again to link them or just copy them.

Chris Ward

Oh crap. Just realized I wasn't using 0.10.5 version of node. Tried to use nvm to switch version but not sure it is completely working. It errors doing a search. I don't think I can find twit that works with 0.10.5. I tried package twit-old, it only uses oauth, but I can't get it to work on my mac.

Steve Fister

@Chris-Ward excellent info! I was trying to narrow it down to a specific module, but hadn't gotten to uuid or har-validate. I did test it in v0.10.0 & v0.10.5 on my Mac and it still continued to work. At least we are on the right track! I'll see if I can't cross compile those two modules for use on this architecture. Thanks!!

Chris Ward

@Steve-Fister What did you use to get v0.10.5 node on your machine? Windows? OSX? Like I said NVM gave me issues. I couldn't find the twit package, and twit-old wasn't working on my mac.

Steve Fister

@Chris-Ward I used NPM on my Mac to get the specific version. I actually downgraded my Mac from the latest version to this one without issue.

Chris Ward

@Steve-Fister yea, ok. I might try that, but I use node for work, and need a different version for that. NVM has worked for me in the past, but I haven't used this old a branch before.

Steve Fister

@Boken-Lin Ok, so I'm having trouble figuring out how to cross-compile a node package (i.e. twit). I've already cross compiled nodejs using JXCore, but can't seem to find any info on doing this for just a package/module. Any help is appreciated!

Boken Lin

Hi @Steve-Fister, We are still trying to figure this out ourselves as well. As soon as we figure out a systematic way to do it, we'll post a tutorial on it.

Currently you just need a package to use the Twitter streaming API? I'm pretty sure there should be a way to use the twitter streaming API without using the request package, which is required by twit and a few other twitter modules for node.js. To solve your immediate problem, I will try to hack together a native-only implementation of the twitter streaming API. Once we have a consistent method to cross-compile node modules, we can then migrate over to existing node modules.

Does that work?

Steve Fister

@Boken-Lin Ok, that's kind of what I figured. Thought I could be Superman, guess not

Yes! If you could figure out a way to get a Twitter stream working in node on the Omega I would OWE YOU BIG TIME!!!!!!