SSH Can’t Connect to Google Cloud Compute Instance

So I go ahead and power up a Google Cloud Compute instance. It doesn’t let me add a key to the VM so I need to wait until the instance is started. Go ahead and add a key. Can’t connect as root, so let’s true their browser ssh. I’m logged in as $myemail@instance. Try to connect as that user? Nope. Using the browser ssh to check out auth.log shows rejection:

Dec  9 17:47:23 instance-1 sshd[3029]: Connection closed by $ip [preauth]

So, I can see that my key has not been copied to ~/.ssh/authorized_keys for $myemail account nor /root/.ssh/authorized_keys . However, a look at /etc/passwd shows that Google has actually created a new unix account, named as the host from the ssh key. Now that the correct user is known, logging in works. Not exactly completely intuitive.

Also, after initially logging into the account and setting up a new “Project”, the following error presented itself:

Error
The project you requested is unavailable.

Then a few hours later the project was able to be manipulated. Also not an insightful message to obfuscate some kind of processing or batched processing occurring.

Read More

A Super Quick Guide to Setting Up a Barebones Puppet Client and Master

This guide covers the set up of a minimalistic Puppet client and master setup on Debian 7.x (similar for Ubuntu 14.x). Guides from Linode and Digitalocean et al. that cover the installation of Puppet seem to feature a lot of deprecated commands, links that 404, unnecessary steps and even the official docs randomly cross into instructions to install a web server (for some reason) and make some other steps annoying and unclear to perform.

 

Machine preparation

We’re going to make use of two machines; one as a client and the other as a master. You can spin up 2 machines on the cloud provider of your choice or make use of a local Virtualbox and Vagrant setup. Ensure the master node is connectible via port 8140. You can test this on the client via “telnet $masterIP 8140″. If it isn’t, fix your firewalling.

You have the option to give the client a DNS address now. This procedure will work if you skip this step, but it means you’ll be generating a certificate request with the clients reverse DNS address, instead.

And we’ll start with this on both nodes:

sudo apt-get update && sudo apt-get upgrade

 
 

Install puppetmaster on Master

Install the puppetmaster package. You can alternatively choose to add the Puppet apt repo here, if you want a newer version than that packaged by your OS.

sudo apt-get install puppetmaster

And stop the daemon;

sudo service puppetmaster stop

 
 

Install puppet on Client

Install the puppet package:

 sudo apt-get install puppet

And stop the daemon:

sudo service puppet stop

 

Generate a CA SSL certificate

If you’re using DNS names to refer to either server, you’ll need to append the FQDN of the master under [main] to /etc/puppet/puppet.conf on the master like so:

dns_alt_names = puppet,yourmaster.domain.com

Otherwise if you’re not setting up DNS, you don’t need to do this. The name ‘puppet’ will be used in this case.

Proceed to generate the CA certificate on master:

sudo puppet master --verbose --no-daemonize

Take care to now wait a few seconds and you’ll see some output saying “Notice: Starting Puppet master version X”. This will look like it’s still doing something but you can now stop the process with a Ctrl+C.
If you get the following error to stdout, this is a packaging issue referencing the non-existent path in fileserver.conf and puppet is thus removing the mount; for our purposes, you can ignore this, create the directory or remove the reference from /etc/puppet/fileserver.conf:

err: Removing mount files: /etc/puppet/files does not exist

 

Add the main manifest

Your puppetmaster process will want a basis to start. Considering we don’t have any modules or configuration in place yet, we’ll create a single config file.

You can ‘touch’ this file, but we’re going to add in some basic config so we can make sure it works. Add the following to: /etc/puppet/manifests/site.pp :

file {'/tmp/testfile':
  ensure  => present,
  mode    => 0755,
  content => "Test file. My eth0 IP is: ${ipaddress_eth0}.",
}

 

Configure the client

We now need to make sure that the client can connect to the master. If you’re using a DNS name to connect to the master, make the following addition to the [main] section in /etc/puppet/puppet.conf on the client:

server = yourmaster.domain.com
certname = yourmaster.domain.com

If you’re not, puppet will need to know how to connect to the master. The Puppet client will try to connect to the host ‘puppet’ by default. As such, you can add the following to /etc/hosts to alleviate (replace $masterIP with your masters IP):

$masterIP yourmaster.domain.com

Test the connection on client:

ping puppet

 

Start service and sign certificate

Start puppetmaster on the master:

sudo service puppetmaster start

Start puppet on the client:

sudo puppet resource service puppet ensure=running enable=true

This is going to generate and send a certificate request to the master.
It’s now wise to change /etc/default/puppet to include “START=yes” and then perform a:

sudo service puppet restart

We’ll now check that the request has been made. On the master, run:

sudo puppet cert list

If you defined a certname on the client earlier, you should see this name appear on this master output. If you didn’t, puppet is going to revert to using the reverse DNS name. We’re now going to approve the request on the master:

sudo puppet cert sign --all

And you should see some output like so:

notice: Signed certificate request for yourmaster.domain.com
notice: Removing file Puppet::SSL::CertificateRequest yourmaster.domain.com at '/var/lib/puppet/ssl/ca/requests/yourmaster.domain.com.pem'

 

Perform a puppet run

The client should now have permissions to connect to the master and solicit config. We’re going to go ahead and perform a puppet run whilst the daemon is still running (-t options include: ‘onetime’, ‘verbose’, ‘ignorecache’, ‘no-daemonize’, ‘no-usecacheonfailure’, ‘detailed-exitcodes’, ‘no-splay’, and ‘show_diff’):

puppet agent -t --verbose

If all is successful, you should see some outline like so:

info: Caching catalog for yourmaster.domain.com
info: Applying configuration version '436345697'
notice: /Stage[main]//File[/tmp/testfile]/ensure: created
notice: Finished catalog run in 0.02 seconds

Check to see if /tmp/testfile exists on the client. If it doesn’t, something has gone wrong.
If the daemon is not active, this should work:

puppet agent --verbose

But trying to launch another daemon with one running would give the error:

Could not prepare for execution: Could not create PID file: /var/run/puppet/agent.pid

Similarly, if the following error appears, you may have some existing cert config that is causing a mismatch. Follow the instructions provided:

Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: .....
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
  puppet cert clean $dnsname
On the agent:
  rm -f /path/to/.pem
  puppet agent -t
Exiting; failed to retrieve certificate and waitforcert is disabled

 
This now completes the installation. 
 

Note that if you see something like the error below, it means the client doesn’t know how to find the master. So either you haven’t defined a “server” in /etc/puppet/puppet.conf or you haven’t given an IP to “puppet” in /etc/hosts :

err: Could not retrieve catalog from remote server: getaddrinfo: Name or service not known
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: getaddrinfo: Name or service not known

 
Also note that if you come across any commands that include “puppetca” or those that aren’t “puppet *” across the interwebs, these are all deprecated commands from Puppt version 2.7 and previous.

Read More

Failure to Clear Android Encryption Message After Formatting and/or Re-flashing

The story is that I had an instance of a Samsung S4 with stock ROM with encryption enabled. I proceeded to flash Cyanogenmod without doing a factory reset first. This worked. A while later I decided to try reflashing some other ROMs and eventually got to a point where the encryption message was still present (and didn’t accept any credentials). The message being “Enter your PIN or password to use the encrypted device memory”, resulting in a reboot loop.

Crazily, flashing many ROMs (which installed fine) or wiping the data/cache and doing a factory reset from Clockworkmod still didn’t rid the message.

Eventually I found a solution in reflashing stock firmware. I originally got some errors, even doing this, through heimdall:

ERROR: libusb error -4 whilst receiving packet. Retrying...

 

ERROR: Failed to confirm end of file

ERROR: RECOVERY upload failed!

 

ERROR: libusb error -4 whilst sending packet. Retrying...

ERROR: Failed to send end session packet!

Releasing device interface...

Re-attaching kernel driver.. 

Eventually, I acquired a Windows box and tried flashing through Odin. This also failed, noting an inability to write. I proceeded to take the sd card out and try again. Success. I Then went through and installed the ROM on the device. However, I was still prompted by the encryption message.

The final step was to then reboot the phone into recovery mode (stock recovery this time.. not Clockworkmod) and perform another factory reset. Then go through Odin and reflash the phone again. Only now did the phone finally boot into an OS.

Then, the sd card can be re-inserted again and the Samsung ROM will handle the formatting for you.

Note to self: If I plan on using encryption and someday flashing another rom, do a damn factory reset before flashing!

Read More

I Don’t Live in Vegas, But I’m Excited for Vegas UberX

I’m no longer in Las Vegas, but am very excited for how Uber can change transport in the city. Why? Because I found the taxi industry in Vegas to be absolutely brutal and a horrible experience for customers. I’ve seen worse taxi industries, but only in places like Mexico and Malaysia, not a US city.

I was in Vegas for just under a week but almost every run-in I had with a taxi driver was poor. A few recounts, based on myself and a group of travellers:

  • Got into a taxi at a casino (very long taxi queues at the casinos) and gave instructions that we’d like to head to another casino further down The Strip. The driver told us that traffic on Las Vegas Blvd. would be slow, so he popped onto an adjacent highway for most of the trip. It is my hypothesis that fast driving like this (and a detour) drove the meter up pretty quickly and a 5 minute ride turned out to be $21. That’s bad enough, but my acquaintance attempted to give the driver $30, who proceeded to pocket the cash as if we weren’t expecting change. We pushed for some cash back, but that’s still kinda dodgy.
  • Approached a taxi while they were stopped at a traffic light coming out of a casino, turning onto The Strip. I asked the driver if he could make a right turn from that position. He proceeded to ask me where I was going. I asked him again if it was possible to make a right turn (which was the direction of a our destination). He responded to this by telling me to “go away!” before waiting another 30 seconds for the light to turn green and driving off without any passengers.
  • With a group of 5 people, we proceeded to find a cab at a rank near Fremont St. (old Strip). Upon realising we were one seat short for a typical sedan carrying 4 passengers, someone in our group uttered “fuck..”. The driver seemed to mis-hear this as “fuck you” [assuming it was directed at him], and diligently responded with a proper “fuck you” and told the occupants of the car to “get out!” before attempting to hail some other customers.

So,  those were a few examples of how I came to have a pretty negative view towards traditional taxis in Las Vegas. I will say that most casinos had very long lines for people wanting a cab, so this allows driver to be very picky with who they pick up because there is never a short supply of customers.

Plus, the aforementioned occurrences are prime examples of the good that can come from Uber and what taxis lack. ie. a feedback system so drivers need to be polite, gps auditing so drivers can’t take the long route and get away with it, all transactions are done through the app so the driver can’t overcharge you at the POS. All very basic stuff.

In regards to Uber, I’m very optimistic about how the new rollout will go. I think they’ll need to be careful to comply with apparent regulation in Las Vegas (an example would be not being able to pick up passengers directly from Las Vegas Blvd (due to traffic problems)). Customers may also need to get used to using the service a little differently, in regards to finding an appropriate location to be picked up and having a way for the driver to locate a passenger in a place with mass amounts of foot traffic.

Read More

How Someone Accessed My Lyft Account and Stole a Ride

If you’ve come to this page wanting tips on cancelling your lyft account or removing a credit card, it’s best to contact Lyft support. These options are not available via the app.*

I’m in California. I’ve spent close to $300 on Uber so far and $0 on Lyft, so I wanted to try it out. After redeeming a first ride coupon, I lined up a great time to try it out. Here is the story of how I lost my first ride.

Upon attempting to try and order a ride, I realised (only after a user clicks through to book a ride) that the app requires a user to add a US contact number to their profile before they can book a ride. It was unfortunate I had to find that out just as I was trying to get somewhere. So I googled the format of US phone numbers and threw a fake number in there. But the app sends a validation message, so I was not be able to validate using this method. As such, I tried setting up Google Voice to get a US number to use for validation, but discovered that even Google requires a US number before you can use their Voice service.

I was out of luck and the processes involved to actually contact Lyft are can be slow (and they want to prevent this, by attempting to limit user messages via FAQs and automation). But even if you navigate through to a contact form, even that requires a US number to submit. And even if I fake my way through that, they could take days to reply.

I googled some more and decided to use a random service online that provides US numbers and a simple public webpage that displays all the text messages sent to these numbers. I used this to get the validation code and activate my account. Great. I was planning to try it out the following morning.

I end up getting an email during the night, which was a ride receipt thanking me for riding with Lyft and documenting my ride. My first ride coupon was used and it would have charged my credit card if I didn’t have that coupon or if the amount surpassed the value of it.

I now have an educated suspicion that instead of logging in via Facebook, someone logged into Lyft using the same publicly available number I had found online. They would have had to validate the login [I assume] via text message, which was easy since they could also see the webpage I had used. They then were able to access my account; see the last 4 numbers of my credit card, my Facebook profile pic and my email address. They then immediately took advantage of this and ordered a ride within locations in Colorado. Image receipt:

Upon seeing this the following morning, I tried to cancel my account but this is not possible without contacting support. I tried to remove my credit card but this is not possible. I proceeded to change my number to a blatantly fake number in the app (after you have validated a first number, Lyft app does not require validation for subsequent numbers). This would hopefully prevent the user from logging into my account again.

In summary, I understand that my own actions caused what happened and that I used the app in a way not intended for users. Still, it was an uncomfortable feeling to find out users could log in via a phone number and that I couldn’t cancel my account, remove my credit card or contact support in an urgent scenario. If the malicious user had decided to change the email address on the account, I can imagine that they could lock me out and then proceed to charge many unrestricted rides to my credit card. I was also so surprised by how quickly this happened (only hours after).

So, I don’t know why Lyft even has the requirement of a US number for accounts. I have used Uber many many times here and never had a problem locating a driver without the use of a phone.

Read More

Android Custom ROM Flashing – Issues

A few issues loading Cyanogenmod onto a Samsung S4 from a Debian 64 bit machine.

Bash would originally fail to execute the adb file (from Android SDK). Even though it was there, bash did not recognise it (with or without sudo:

./adb bash: ./adb: No such file or directory

Surprisingly enough, it was a problem with missing dependencies. Many people recommended installing ia32-libs but the fix for me were the following packages:

sudo apt-get install libc6-i386 lib32stdc++6 lib32gcc1 lib32ncurses5

Next, I was using adb sideload to copy my rom, rather than pushing it or copying it via any other method. I was getting:

ADB: error: insufficient permissions for device

Fix was to [when the device is connected]: “sudo ./adb kill-server”, “./adb start-server” and “./adb sideload /path/to/rom.zip”.

Next, I also was unable to properly use Cyanogenmod after I flashed the ROM. It was taking me to a “Encryption unsuccessful” message and while it offered a button to attempt to fix it, this then resulted in a reboot loop [if you kept pressing the buttom]. This would relate to the fact I had encrypted my S4 and sd card in the past.

This seemed tricky to fix (after trying a few things and wanting to tread carefully) but I ended up rebooting into recovery once more and playing around. Most of the options in ClockworkMod Recovery [formatting-wise], failed with an error. Attempting a “format /data” failed, but for some reason, I had luck with the following option, with proceeded to *actually* format the partition for me and fix the problem:

 format /data and /data/media (/sdcard)

I then proceeded to re-run the original options to “wipe data/factory reset”, “wipe cache partition” and within “Advanced”, “wipe dalvik cache”. Then repeat the step of “install zip”, “install zip from sideload” and try again. I was then able to boot into Cyanogenmod fo’ real.

Read More

OpenNMS Failing to Start

This will outline some troubleshooting steps to take when OpenNMS is refusing to start. In this scenario, the service looked like the following, before eventually stopping:

OpenNMS.Eventd         : start_pending
OpenNMS.Trapd          : start_pending
OpenNMS.Queued         : start_pending
OpenNMS.Actiond        : start_pending
OpenNMS.Notifd         : start_pending
OpenNMS.Scriptd        : start_pending
OpenNMS.Rtcd           : start_pending
OpenNMS.Pollerd        : start_pending
OpenNMS.PollerBackEnd  : start_pending
OpenNMS.Ticketer       : start_pending
OpenNMS.Collectd       : start_pending
OpenNMS.Discovery      : start_pending
OpenNMS.Vacuumd        : start_pending
OpenNMS.EventTranslator: start_pending
OpenNMS.PassiveStatusd : start_pending
OpenNMS.Statsd         : start_pending
OpenNMS.Provisiond     : start_pending
OpenNMS.Reportd        : start_pending
OpenNMS.Alarmd         : start_pending
OpenNMS.Ackd           : start_pending
OpenNMS.JettyServer    : start_pending
opennms is partially running
[00:11:56]-> service opennms status
Could not connect to 127.0.0.1 on port 8181 (OpenNMS might not be running or could be starting up or shutting down): Connection refused
opennms is stopped

With the web server rendering:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

A key step is going to be analysing the daemon logs to assess any potential issues, which can be found in: /var/log/opennms/daemon/ (tail -f /var/log/opennms/daemon/* to show them in real-time, as you start the service in another terminal).

You may see some exceptions like the following, but they are commonly only shown as a symptom of another problem. AKA. it is unable to call stop because the service could never start:

2014-06-15 00:19:29,166 DEBUG [Main] Invoker: Invoking stop on object OpenNMS:Name=Vacuumd
2014-06-15 00:19:29,172 ERROR [Main] Invoker: An error occurred invoking operation stop on MBean OpenNMS:Name=Vacuumd: javax.management.RuntimeMBeanException: java.lang.NullPointerException
javax.management.RuntimeMBeanException: java.lang.NullPointerException
2014-06-15 00:19:29,231 DEBUG [Main] Manager: Thread dump completed.
2014-06-15 00:19:29,232 DEBUG [Main] Manager: memory usage (free/used/total/max allowed): 47401760/116815072/164216832/1200160768
2014-06-15 00:19:29,232 INFO  [Main] Manager: calling System.exit(1)
An error occurred while attempting to start the "OpenNMS:Name=Notifd" service (class org.opennms.netmgt.notifd.jmx.Notifd).  Shutting down and exiting.
javax.management.RuntimeMBeanException: java.lang.reflect.UndeclaredThrowableException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)

However, this is indicative of a startup problem and a common reason for this is broken config files. A good step would be to have a close look at any files yourself or others have modified and perform a check on them. Else, you can parse the XML for syntax errors using xmllint:

xmllint --noout /etc/opennms/*xml

At least in my case, this showed that there was a missing “<" at the beginning of a file I had been working on, but had somehow accidentally removed this character.

notifd-configuration.xml:1: parser error : Start tag expected, ‘<' not found
?xml version="1.0" encoding="UTF-8"?>
^

Fix this up and perform another service start on OpenNMS. It can take a while to start but you can keep checking the status to see if the service remains active until it eventually gets into a permanent working state.

Read More

Salt Failing to Highstate due to Keys on Upgrade to 2014

Upon upgrading from a 17.x to 2014 version of Saltstack, some care needs to be taken to switch out the keys in the process. “Deleting” the key (with ‘salt-key -D’) will not be sufficient on its own.

Upon upgrading an existing instance of a Salt Master, you may notice issues when attempting to highstate a Minion (even a Minion and Master on the same box). Running the highstate in debug mode will identify delay and repeated attempts to load in keys.

[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

The solution is to delete everything within /etc/salt/pki/* on Master and Minion. Then, delete the existing key from the Mater (salt-key -D or salt-key -d $key). Proceed to restart Minions and Master. You can test connectivity via “sudo salt \* test.ping” and proceed with another highstate to see if the problem remains.

Read More

Apache Redirect to Alternate Domain Whilst Passing URI

Moving a site such as domain.com/oldblog to domain.com/blog using Apache/Apache2. The requirements were to maintain the URI and 301 for SEO purposes. The last rule will carry across the URI, so hitting domain.com/oldblog/something/wat.jpg will forward through to domain.com/blog/something/wat.jpg . The first rule is to catch it when you hit domain.com/oldblog directly. The second query catches the trailing slash, so redirects when you hit domain.com/oldsite/ . The lack of “L” in the first two rules tells the web server that this is not the final rule for this pattern, and to continue parsing other rewrites.

RewriteRule ^/oldblog$ /blog [R=301]
RewriteRule ^/oldblog/$ /blog [R=301]
RewriteRule ^/oldblog.* /blog%{REQUEST_URI} [R=301,L]

Read More

Cloudflare and Transfer-Encoding of Absent or Chunking in a 403 Java App

There was a problem where a java application was returning this error, after a recent switch to Cloudflare:

Application Error
Application Error	 	A general application error has occurred. 
(java.io.IOException) 
Server returned HTTP response code: 403 for URL: https://url.com:443/somepath.xml

It was noticed that the HTTP response header of “Transfer-Encoding:chunked” through Cloudflare, where it was otherwise present when hitting the web server directly.

One option was to look into Cloudflares different levels of caching. They have basic, simplified and aggressive.

Transfer-encoding denotes the transfer method used by HTTP to transfer data to the user. It can be chunked etc.

It turns out the problem was the Cloudflare Web Application firewall was adding in rules to block IPs. Due to this feature being hard to find in the current design of their website, you can find and configure it here: https://www.cloudflare.com/waf . Please note this is only for paid accounts.

Read More