Thursday, November 17, 2016

Delete Git Branch Local and Remote

Executive Summary

$ git branch -d 
$ git push origin --delete 

Delete Local Branch

To delete the local branch use:
$ git branch -d branch_name
Note: The -d option is an alias for --delete, which only deletes the branch if it has already been fully merged in its upstream branch. You could also use -D, which is an alias for --delete --force, which deletes the branch "irrespective of its merged status." [Source: man git-branch]

Delete Remote Branch [Updated on 1-Feb-2012]

As of Git v1.7.0, you can delete a remote branch using
$ git push origin --delete 
which might be easier to remember than
$ git push origin :

Friday, September 02, 2016

Gradle Dependencies

https://docs.gradle.org/current/userguide/tutorial_gradle_command_line.html
gradle -q api:dependencies --configuration testCompile

Wednesday, August 17, 2016

Externalize Common Gradle Functions

Content of helpers/common-methods.gradle:

// Define methods as usual
def commonMethod1(param){
    return true
}
def commonMethod2(param){
    return true
}

// Export methods by turning them into closures
ext{
    commonMethod1 = this.&commonMethod1
    otherNameForMethod2 = this.&commonMethod2
}
And this is how I use those methods in another script:

// Use double-quotes, otherwise $ won't work
apply from: "http://myhelpers.com/helpers/common-methods.gradle"

// You can also use URLs
//apply from: "https://bitbucket.org/mb/build_scripts/raw/master/common-methods.gradle"

task myBuildTask{    
    def myVar = commonMethod1("parameter1")
    otherNameForMethod2(myVar)    
}

Thursday, August 04, 2016

Trust Relationship and Policies for AWS API Gateway and Lambdas

Your Policy for the lambda should set up everything your lambda is allowed to do. This includes passing a role.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1470153553000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-west-2:439753510372:table/YoYoDyne_Products"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        },
        {
            "Sid": "Stmt1449789105000",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
Your policy also needs to have a trust relationship.
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "",
           "Effect": "Allow",
           "Principal": {
               "Service": ["lambda.amazonaws.com", "apigateway.amazonaws.com"]
           },
           "Action": "sts:AssumeRole"
       }
   ]
}

Tuesday, August 02, 2016

Java HTTPS Over VPN: Unrecognized SSL message, plaintext connection?

Many times a VPN will screw around with the IPV4 settings for secure connections.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
    at com.sun.net.ssl.internal.ssl.InputRecord.handleUnknownRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
Add these VM options to avoid problems:
-Djsse.enableSNIExtension=false
-Djava.net.preferIPv4Stack=true

Wednesday, June 15, 2016

Find What Process is Running on a Port OSX

17:12:52 ~/data/github/dronze {feature/#58} $ lsof -i :8000
COMMAND   PID          USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java    29680 claytongraham  143u  IPv6 0xb9a0c6ac4b96e77f      0t0  TCP *:irdmi (LISTEN)

Monday, June 13, 2016

Minimum Mongo Restore Permissions

So this would work to restore a single database: Backup
sudo mongodump -h localhost -d stuffdb --username restoreSingleDB --password Moon1234 \
  -o stuffdb.20160724.dump
Restore
db.addUser( {
    user: "restoreSingleDB",
    pwd: "Moon1234",
    roles: [ "readWrite", "userAdmin" ]
} )

mongorestore --db  --username restoreSingleDB --password Moon1234 /
So this would work to restore all databases including user data:
db.addUser( {
    user: "restoreAll",
    pwd: "Moon1234",
    roles: [ "readWriteAnyDatabase", "userAdminAnyDatabase" ]
} )

mongorestore --username restoreAll --password Moon1234 /

Sunday, June 05, 2016

Documenting a REST API

I have found that MkDocs is pretty good. I can integrate it into a CICD build with little effort. What was not obvious was best practices for writing API docs. So I create a template! Check it out:

Tickers V1


Gives the user access to the available tickers.

Path: /rest/v1/tickers

GET /tickers/ticker/{ticker}

@PreAuthorize("hasAuthority('api')")
@RequestMapping(path = "/ticker/{ticker}",
    method = RequestMethod.GET,
    produces = MediaType.APPLICATION_JSON_VALUE )

Params

Param Type Required Notes
ticker path YES

Info

Attribute Value Notes
hasAuthority api Must be an api user to make calls.
jwt Bearer Call must have valid bearer token.
consumes NONE
produces application/json Generates JSON response.

curl example

curl -X GET -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0NjUxNzQyNDMsImF1dGhvcml0aWVzIjpbImFwaSJdLCJzdWJqZWN0IjoiMTE0OTEwOTg2ODIzOTczNDcwMTE0QGdvb2dsZSIsImlhdCI6MTQ2NTE3MDY0M30._cTZ_8rp0Z19cvzAGhXsNqD879KL9rAp17TWI3Qs6mk" 
-H "Cache-Control: no-cache" "https://api.dronze.com/rest/v1/tickers/ticker/AAPL"

response

{
  "id": "569f2d8b67d02fd037e5cd5e",
  "type": "ticker",
  "labels": [
    "AAPL",
    "APPLE INC"
  ],
  "embedded": {
    "currency": "USD",
    "cusip": "037833100",
    "exchange": "NASDAQ",
    "industry": "Electronic Equipment",
    "isin": "US0378331005",
    "last_updated": "Sep 25, 2015 12:00:00 AM",
    "perma_ticker_id": 199059,
    "prior_tickers": "None",
    "related_tickers": "None",
    "sic": 3571,
    "ticker": "AAPL",
    "ticker_name": "APPLE INC",
    "ticker_sector": "Technology"
  }
}
view raw example_rest.md hosted with ❤ by GitHub

Subdomain CNAME with Cloudfront

Automation is good. It means less mistakes and simple cookbooks for runbooks and deployments. Here is a simple recipe for deploying a subdomain CNAME with cloudfront.
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudFormation Template Route53_CNAME. This uses the patameters to create a subdomain CNAME",
"Parameters" : {
"HostedZone" : {
"Type" : "String",
"Description" : "The DNS name of an existing Amazon Route 53 hosted zone",
"AllowedPattern" : "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
"ConstraintDescription" : "must be a valid DNS zone name."
},
"SubdomainParam": {
"Type": "String",
"Description": "Enter the subdomain name for the stack you want mapped to the CNAME"
},
"LoadBalancerParam": {
"Type": "String",
"Description": "Enter the loadbalancer name for the stack you want mapped to the CNAME"
}
},
"Resources" : {
"SubdomainDNSRecord" : {
"Type" : "AWS::Route53::RecordSet",
"Properties" : {
"HostedZoneName" : { "Fn::Join" : [ "", [{"Ref" : "HostedZone"}, "." ]]},
"Comment" : "CNAME redirect to subdomain.",
"Name" : { "Fn::Join" : [ "", [{"Ref" : "SubdomainParam"}, ".", {"Ref" : "HostedZone"}, "."]]},
"Type" : "CNAME",
"TTL" : "900",
"ResourceRecords" : [{
"Ref": "LoadBalancerParam"
}]
}
}
},
"Outputs" : {
"CNAME" : {
"Description" : "Fully qualified domain name",
"Value" : { "Ref" : "SubdomainDNSRecord" }
}
}
}
#!/usr/bin/env bash
aws cloudformation create-stack --profile default --stack-name stage-blog-cname \
--template-body file:///Users/claytongraham/data/github/dronze/dronze-cicd/cf/r53_cname.json \
--parameters ParameterKey=HostedZone,ParameterValue=dronze.com \
ParameterKey=SubdomainParam,ParameterValue=stage.blog \
ParameterKey=LoadBalancerParam,ParameterValue=dronze-elb-PROD-1699664721.us-west-2.elb.amazonaws.com
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "cloudformation:*",
"Effect": "Allow",
"Resource": "arn:aws:cloudformation:us-west-2:715212546939:stack/*"
},
{
"Action": "route53:*",
"Effect": "Allow",
"Resource": "arn:aws:route53:::hostedzone/Z3VRQF6A9GQX5I"
},
{
"Action": "codepipeline:*",
"Effect": "Allow",
"Resource": "*"
}
]
}

Saturday, May 21, 2016

Upload HTTPS Cert to Cloudfront

23:28:23 ~/data/dronze/keys $ aws iam upload-server-certificate \
   --server-certificate-name dronze.com --certificate-body file://dronze_com_crt.pem \
   --private-key file://dronze_com.key --certificate-chain file://dronze_com.ca-bundle \
   --path /cloudfront/production/
{
    "ServerCertificateMetadata": {
        "ServerCertificateId": "ASCAIDAGVOZBPZ6VJTK7A",
        "ServerCertificateName": "dronze.com",
        "Expiration": "2017-05-22T23:59:59Z",
        "Path": "/cloudfront/production/",
        "Arn": "arn:aws:iam::705212546939:server-certificate/cloudfront/production/dronze.com",
        "UploadDate": "2016-05-22T06:30:39.224Z"
    }
}

Tuesday, May 03, 2016

How to set your log4j2 file to support the lambda appender.

Add to your gradle config:

compile 'com.amazonaws:aws-lambda-java-log4j:1.0.0'
and How to set your log4j2 file to support the lambda appender.


<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" packages="com.amazonaws.services.lambda.runtime.log4j">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level %C - %msg%n"/>
</Console>
<LambdaAppender name="LambdaAppender">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n"/>
</LambdaAppender>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Console" />
</Root>
<Logger name="com.dronze.aws.lambda" level="debug" additivity="false">
<AppenderRef ref="LambdaAppender" />
</Logger>
</Loggers>
</Configuration>

Monday, May 02, 2016

Allowing profiles to be passed to bootRun

Allowing profiles to be passed to bootRun

bootRun {
addResources = false
//pass command line args
if ( project.hasProperty('jvmArgs') ) {
jvmArgs project.jvmArgs.split('\\s+')
}
}
view raw bootRun.gradle hosted with ❤ by GitHub

$ ./gradlew bootRun -PjvmArgs="-Dspring.profiles.active=docker"

Spring Profiles

Spring Profiles

Spring Profiles provide a way to segregate parts of your application configuration and make it only available in certain environments. Any@Component or @Configuration can be marked with @Profile to limit when it is loaded:
@Configuration
@Profile("production")
public class ProductionConfiguration {

    // ...

}
In the normal Spring way, you can use a spring.profiles.active Environment property to specify which profiles are active. You can specify the property in any of the usual ways, for example you could include it in your application.properties:
spring.profiles.active=dev,hsqldb
or specify on the command line using the switch --spring.profiles.active=dev,hsqldb.

Saturday, April 30, 2016

Thursday, April 21, 2016

Installing Amazon Web Service Plugin for ElasticSearch

https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws.html

The Amazon Web Service (AWS) Cloud plugin uses the AWS API for unicast discovery, and adds support for using S3 as a repository for Snapshot/Restore.

Installationedit

This plugin can be installed using the plugin manager:
sudo bin/plugin install cloud-aws
-> Installing repository-s3...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/repository-s3/2.3.1/repository-s3-2.3.1.zip ...
ERROR: failed to download out of all possible locations..., use --verbose to get detailed information
# bin/plugin install cloud-aws
-> Installing cloud-aws...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/cloud-aws/2.3.1/cloud-aws-2.3.1.zip ...
Downloading .............................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/cloud-aws/2.3.1/cloud-aws-2.3.1.zip checksums if available ...
Downloading .DONE
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission getClassLoader
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed cloud-aws into /usr/share/elasticsearch/plugins/cloud-aws
# service elasticsearch restart
[....] Stopping Elasticsearch Server:

Tuesday, April 19, 2016

Spring Boot CORS Not Working?

For the life of me I could not get the CORS Configuration for Spring Boot to write the needed headers to the response. I wrote my own filter that could read the CORS Config and used it in WebMVC Configuration to solve it.

package com.dronze.app;
import com.dronze.filter.CorsFilter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration;
import org.springframework.boot.context.embedded.EmbeddedServletContainerFactory;
import org.springframework.boot.context.embedded.FilterRegistrationBean;
import org.springframework.boot.context.embedded.ServletRegistrationBean;
import org.springframework.boot.context.embedded.jetty.JettyEmbeddedServletContainerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
import org.springframework.web.servlet.DispatcherServlet;
/**
* Created by claytongraham on 10/25/15.
*/
@Configuration
@ComponentScan(basePackages = {"com.dronze"})
@PropertySource("classpath:/application-${dronze.env:dev}.properties")
public class AppConfig {
@Bean
public EmbeddedServletContainerFactory servletContainer() {
JettyEmbeddedServletContainerFactory server = new JettyEmbeddedServletContainerFactory();
return server;
}
@Bean
public DispatcherServlet dispatcherServlet() {
return new DispatcherServlet();
}
@Bean
public ServletRegistrationBean dispatcherServletRegistration() {
ServletRegistrationBean registration = new ServletRegistrationBean(
dispatcherServlet(), "/rest/api/*");
registration.setName(DispatcherServletAutoConfiguration.DEFAULT_DISPATCHER_SERVLET_REGISTRATION_BEAN_NAME);
return registration;
}
@Bean
public FilterRegistrationBean corsFilter() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.setAllowCredentials(true);
config.addAllowedOrigin("*");
config.addAllowedHeader("Content-Type");
config.addAllowedMethod("OPTIONS");
config.addAllowedMethod("GET");
config.addAllowedMethod("PUT");
config.addAllowedMethod("POST");
config.addAllowedMethod("DELETE");
source.registerCorsConfiguration("/**", config);
final FilterRegistrationBean bean = new FilterRegistrationBean(new CorsFilter(source));
bean.setOrder(0);
return bean;
}
}
view raw AppConfig.java hosted with ❤ by GitHub
package com.dronze.filter;
/**
* Custom Filter Created by claytongraham on 4/19/16.
*/
import com.amazonaws.util.StringUtils;
import org.springframework.http.HttpHeaders;
import org.springframework.util.Assert;
import org.springframework.web.cors.*;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
import java.util.*;
public class CorsFilter extends OncePerRequestFilter {
public static final String CUSTOM_ALLOW_CORS_HEADER = "Accept-Cors-Requests-Origin";
private CorsProcessor processor = new DefaultCorsProcessor();
private final CorsConfigurationSource configSource;
/**
* Constructor accepting a {@link CorsConfigurationSource} used by the filter to find
* the {@link org.springframework.web.cors.CorsConfiguration} to use for each incoming request.
* @see org.springframework.web.cors.UrlBasedCorsConfigurationSource
*/
public CorsFilter(CorsConfigurationSource configSource) {
this.configSource = configSource;
}
/**
* Configure a custom {@link CorsProcessor} to use to apply the matched
* {@link org.springframework.web.cors.CorsConfiguration} for a request.
* <p>By default {@link DefaultCorsProcessor} is used.
*/
public void setCorsProcessor(CorsProcessor processor) {
Assert.notNull(processor, "CorsProcessor must not be null");
this.processor = processor;
}
public static boolean isCorsRequest(HttpServletRequest request) {
return (!StringUtils.isNullOrEmpty(request.getHeader(HttpHeaders.ORIGIN)));
}
private String getCSVList(List<String> items){
String[] array = (String[])items.toArray(new String[items.size()]);
return StringUtils.join(",", array);
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
CorsConfiguration corsConfiguration = configSource.getCorsConfiguration(request);
if(isCorsRequest(request)){
response.addHeader("Access-Control-Allow-Origin", getCSVList(corsConfiguration.getAllowedOrigins()));
response.addHeader("Access-Control-Allow-Headers", getCSVList(corsConfiguration.getAllowedHeaders()));
response.addHeader("Access-Control-Allow-Methods", getCSVList(corsConfiguration.getAllowedMethods()));
}
filterChain.doFilter(request, response);
}
}
view raw CorsFilter.java hosted with ❤ by GitHub

Saturday, April 16, 2016

Find Recursively With LS

This should probably be turned into a script.
[claytongraham@Claytons-MacBook-Pro dronze]$ find . -name '*.jar' -exec ls -lah {} \;
-rw-r--r-- 1 claytongraham staff 261B Apr 16 11:39 ./build/libs/dronze-1.0.jar
-rw-r--r-- 1 claytongraham staff 51K Apr 6 21:13 ./dronze-app/gradle/wrapper/gradle-wrapper.jar
-rw-r--r-- 1 claytongraham staff 50K Apr 16 11:39 ./dronze-cicd/build/libs/dronze-cicd.jar
-rw-r--r-- 1 claytongraham staff 51K Apr 6 21:13 ./dronze-cicd/gradle/wrapper/gradle-wrapper.jar
-rw-r--r-- 1 claytongraham staff 261B Apr 16 11:39 ./dronze-config/build/libs/dronze-config-0.3.1-SNAPSHOT.jar
-rw-r--r-- 1 claytongraham staff 51K Apr 6 21:13 ./dronze-config/gradle/wrapper/gradle-wrapper.jar
-rw-r--r-- 1 claytongraham staff 51K Apr 6 21:13 ./gradle/wrapper/gradle-wrapper.jar
view raw findwls.bash hosted with ❤ by GitHub

Monday, April 11, 2016

Rsync Without Git

Just add an explicit exclude for .git:
rsync -a --exclude='.git/' --include='*.c' --include='*.sh' --include='*/' --exclude='*' ~/c/ ~/Dropbox/Public/c
Another option is to create ~/.cvsignore containing the following line along with any other directories you'd like to exclude:
.git/

Tuesday, February 23, 2016

Thursday, February 18, 2016

Find In Files Recursively

Its helpful to be able to search for a string recursively in a directory. Here is the basic way:

grep -rnw '/path/to/somewhere/' -e "pattern"
-r or -R is recursive, -n is line number and -w stands match the whole word. -l (lower-case L) can be added to just give the file name of matching files.
Along with these, --exclude or --include parameter could be used for efficient searching. Something like below:
grep --include=\*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"
This will only search through the files which have .c or .h extensions. Similarly a sample use of --exclude:
grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"
Above will exclude searching all the files ending with .o extension. Just like exclude file it's possible to exclude/include directories through --exclude-dir and --include-dir parameter; for example, the following shows how to integrate --exclude-dir:
grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"


Replace Recursively


find /home/www -type f -print0 | xargs -0 sed -i 's/subdomainA.example.com/subdomainB.example.com/g'

Wednesday, February 17, 2016

Truncating Git History

Look, sometimes you want to be a revisionist, and the only way to get rid of a pesky diff that is taking a lot of commit time is to truncate the commit log. So in the rare occasion you need to do it here is how.

Create a Truncate Script

#!/bin/bash
git checkout --orphan temp $1
git commit -m "First commit"
git rebase --onto temp $1 master
git branch -D temp
view raw git-truncate.sh hosted with ❤ by GitHub

Use the Commit Sha to Truncate To

source ~/data/bin/git-truncate a71f4386f9d5e3b707067e66fd5f76a1cc06c11c

Force the Push to Origin

git push origin master --force

Thursday, January 21, 2016

Basic Screen Commands

Screen is an extremely useful tool. When you dont want to make a service and just need something running in the background. Here are the basic screen commands.

#create a screen bitnami@ip-172-31-53-17:~$ screen -l

#detach a screen ^A + D

#list screens

bitnami@ip-172-31-53-17:~$ screen -list
There are screens on:
	846.pts-11.ip-172-31-53-17	(01/21/2016 09:25:13 PM)	(Detached)
	409.pts-8.ip-172-31-53-17	(01/21/2016 09:22:30 PM)	(Attached)
	22660.pts-9.ip-172-31-53-17	(01/04/2016 07:29:41 PM)	(Detached)
	22643.pts-6.ip-172-31-53-17	(01/04/2016 07:29:02 PM)	(Detached)

#remove a detached session

bitnami@ip-172-31-53-17:~$ screen -X -S 22660.pts-9.ip-172-31-53-17 quit