Book Review: America the Vulnerable: Inside the New Threat Matrix of Digital Espionage, Crime, and Warfare

I guess I am in like a book posting series for now but since I had to do a book review to earn CPE credits for my CISSP certification I thought it would get a cheap blog entry this month with it.

This book was a great follow up to Fatal Systems Error for me. Joel Brenner, the author, is a former Inspector General of the National Security Agency, NSA, and was also the head of U.S. Counterintelligence for the director of National Intelligence.  This book explores both Cybercrime and Cyberwarfare with more emphasis in the latter. The author does give the reader a primer on Cybersecurity and how some attacks such as DDoS for example take place. However, because of Brenner’s insight into the US national intelligence system, the book focuses on what State-sponsored Cyberthreats represents to the international community, especially so to The US without going into much detail into how attacks happen. It describes how Cybercrime has evolved as a potential lethal weapon to a nation’s critical infrastructure (e.g. power, financial, communications, and military industries) by using the internet to disturb communications and information flow during warfare and references to how Serbians manipulated these tools during the Kosovo War; most recently you could probably relate to the Egypt Revolution and the critical role that communications and Social Media played there, and continues threats by the Iranian government to shut down communications in their country as a way to control and intimidate their opponents. When The People´s Republic of China, PRC, realized that they were too far behind from The US economics and military forces they identified an opportunity to explore other ways to build their technology and industries by exploiting the weaknesses of our current industries and military forces to steal classified information and intellectual property that they have used to catch up with the our technology and use it against us. Both government and the private sector are at fault here by not implementing the necessary security control and most importantly by failing to educate and create security awareness within their organizations. 

The moment of truth for me during this read was when I came acrros the excerpt below. The reason being that I work in a mid-size law firm and I couldn’t agree more with this statement which actually comes from a lawyer and partner in one of the largest 100 law firms in the US.

"The Chinese have other even subtler methods of stealing our know-how. Several years ago, while serving as the national counterintelligence executive, I sat with colleagues discussing how we would plan an espionage attack against an American business. And then a lightbulb went on: the law firms! Of course: A company’s outside intellectual property lawyers have its technical secrets, and their corporate law colleagues are privy to strategic business plans. And lawyers don’t like taking instructions from anybody, particularly their less well paid underlings who are responsible for network security. They’re impatient. In some firms the rainmakers have nixed even simple steps, like requiring a password on mobile devices that connect with the firm’s servers. They couldn't be bothered. Privileged with secrets, highly paid, often arrogant and usually impatient, lawyers are the perfect targets. I cannot disclose what I know because it's classified, but I can disclose that I know that my surmise was soon justified. U.S. law firms have been penetrated both here and abroad. Firms with offices in China and Russia are particularly vulnerable, because the foreign security services are likely to own the people who handle the firms' physical and electronic security. These services are not interested in stealing brilliant legal briefs; they want information about the firm's clients. Every law firm with offices in several continents holds privileged and sensitive electronic documents worth millions of dollars to a foreign service, ranging from investment plans to negotiating and business strategies, and much more." Think I got hooked up here? No kidding! This is what I am dealing with every day and is such a great challenge and opportunity for me as a professional. If you are in legal and have kept up with the news then you now know what Brenner knew. There are plenty of articles out there on how Chinese are attacking firms, security in law firms, or recent incidents such as Anonymous attacking law firms. Dear lawyers, this is a real issue for us and you guys need to pay attention.

Brenner also does a great job on identifying one of the most important problems that governments face: finding balance between transparency and privacy. While transparency says “Open Up”, privacy says “We are watching you and you are very restricted to what we want you to do.” Both transparency and privacy are about information which he refers to as being liquid. I love that analogy because information as liquid can take any form and be anywhere. The problem is that once liquid leaks, it is hard to figure out how, when and where it happened, not to mention that it is nearly impossible to recover it all. During this segment the author also references how organizations such as WikiLeaks operate by “turning the hose on” to let the precious liquid out. The appearance of WikiLeaks was a critical milestone to the proliferation of Hactivist groups such as Anonymous which supported them by launching DDoS to organizations such as PayPal and MasterCard that froze WikiLeaks funds in an attempt to stop their operations. It is important to note that WikiLeaks has a more organized structure governed by their decision maker Julian Assange, while hactivist groups do not have any type of hierarchy or governing entity and function as a group of people that support the same “values or ideas”.

In chapter 7, the author presents a hypothetical scenario where he illustrates how China could potentially create serious damage and pretty much “own” The US on the verge of war using techniques such as shutting down entire power grids in the US, sending off undetectable submarines to face The US Navy crafts and disturbing the financial markets. He actually mentioned that while his scenario is fictional, some of the penetrations and techniques used by the PRC have actually already happened.

Finally, Brenner presents what he believes are good practices to get both the public and private sectors together. Many of them are widely known, yet not practiced much. Here they are in a nutshell:

Public Sector:

1. Stronger trade regulations and contracting. Requiring higher security standards from its vendors.
2. Make Service Providers accountable. For example require ISP to notify customers whose machines are infected by a botnet.
3. Stronger Energy Standards. Limit connectivity to a public network.
4.  Tax code. Use tax incentives to encourage investment in cybersecurity
5. Encourage and found research.
6. Securities regulations
7. International relations. International community needs to come together in all of these efforts.

Private Sector:

1. Clean up your act! Assume that you have been attacked so monitor and mitigate.
2. Control WHAT is in your system.
3. Control WHO is in your system.
4. Protect what is valuable.
5. Patch, patch, patch.
6. Train, train, train.
7. Audit for operational effect.
8. Manage overseas travel behavior.

2011: A Year Of Learning

Hello everyone. I feel like I have abandoned my blog but I have been quite busy lately working in a few projects. I have been writing though, with two blog entries and two articles for the International Legal Technology Association, ILTA.

What I want to accomplish with this post is not only summarize the highlights of my year, in terms of my career, but also what I learned from each project and situation and how I grew as consequence of them. So here I go.

Writing

I thought it was appropriate to start with this blog. It was launched in February with a goal of posting at least once a month but most importantly adding value. I did pretty well until the end of the year, when I got busy with our Windows 7 Roll out which I actually blogged about here and here. I feel like I continued to improve and define my writing style, and I learned how powerful social networks could really be. I never thought that I would reach over 1ooo people in three different continents with just a few posts and I hope I am adding value. Furthermore, I confirmed something that I know and practice; not taking action in your ideas and projects is a bad habit. Keep posting is a commitment that I made to myself and I am happy with the results primarily because I also participated in our ILTA Connected Community blog, but there are a couple of entries that I started and I think that would have been great and timely but never finished. So the take away is that once we have an idea or commit to something we must take action and finish it.

Many of you know that I serve as the Servers Operations & Security Peer Group VP for the International Legal Technology Association, ILTA and we launched a new blog which we hope will add value to our members. You can read not only our posts, but other formidable entries regarding Legal Technology here. In addition I wrote an article on Network Security for ILTA’s Risky Business White Paper and another for their Peer to Peer Magazine which describes how I work on strengthening my mind to achieve results and goals which you can read here. Again, I was reminded that I really need to make sure that I know and understand what I am writing about, which is the case, because of the impact that it could have in others. I re-affirmed that the goal is to add value.

Technology

I was pretty busy at Nexsen Pruet early in the year with infrastructure projects that included enhancing our dual MPLS network architecture by adding a second router to each location (formerly dual-homed in a single router). I learned more about BGP and EIGRP routing during this project and that while our network got more robust, it also did get more complicated. I also learned that Disaster Recovery and Business Continuity (DRBC) is an ongoing process that needs to be tested and adjusted properly as network and system topologies evolve. Besides, I again confirmed the importance of keeping good network and systems documentation up to date.

Four other major projects took place: an upgrade to our Video Conference infrastructure (VC), which I posted about here, the introduction of Application Level Monitoring through Riverbed’s Cascade appliance, expansion of Mobile Device fleet by introducing iPhones and MobileIron as our MDM solution, and a merger. I learned that while our VC system was heavily used the quality of experience by both the participants and IT folks in our team was very poor. Yes the investment was hefty, but the user experience has improved dramatically and in the last 2-3 months the adoption of desktop video both internally and externally has grown and is already in demand for 2012. That in addition to the ease of management tools of the new systems has probed the investment worthy.

We now have a better idea of what is going on in our network as far of traffic. We had been using Netflow collectors for a while which gave us a good picture of who was doing what, but I now personally have a much stronger understanding of how our applications interact with users and each other. Cascade gives us a great picture of all pieces interacting in a user action within our network and when there is a problem, it could tell us where it is, whether it is the network, servers, or clients. We are still on learning mode with this tool.

The introduction of iPhones and a merger kept me busy for a while. Whereas I learned a lot about mobile technologies and enhanced my knowledge of general security practices, I also discovered one a new passion on these two projects: Project Management. I have been managing projects since I became the Network Manager at the firm especially around telephony and facilities having c0-managed the last office move from a technology standpoint. The reason I like PM and that I will make an effort to continue to learn about this topic is because it gives me skills outside IT that I feel I need in order to continue to advance my career, specially so Risk Management, an area of great interest to me at the moment. I am not at a PMP level yet but I think that I am doing well and in fact, I have been tasked with building our Technology Project Management practice at the firm and now have a very talented PM reporting to me.

The rest of the year was dedicated to our Windows 7 and Office 2010 rollout. I learned so much that I don’t know where to start; from how many specialty applications we are running, to how attorneys and their staff work and use those apps, to how successful you could be when everyone is on board and rowing together towards the same goal. This was the most rewarding, and one of the most successful projects that I have been part of, and again, project management and planning was the key. My role during the project was of Deployment Manager in two different offices and Project Manager in two others. I learned a lot about how attorneys at different offices and practicing different laws work, and how unique their practices are. The most satisfying part of the project though, was seeing how so many talented people in our Technology Department grew during this project. This project also let me understand how important making adjustments while staying focus is; things go wrong, get delayed or even completely pushed out of the initial plan, but you must remain focused on the end goal and adjust as you go, which means that you must evaluate where you are constantly.

Leadership

I will never want to stop learning on this area, period. It is such a broad and complex, yet intriguing topic that I think evolution is the key to its mastering, if there is such thing. Anyway, as mentioned earlier I lead a group of legal technology professionals for ILTA. I got appointed as the VP for the Servers Operations Peer Group in August. One thing that I learned firsthand is how important succession is. My predecessor, Bob DuBois, did a great job getting me ready for the transition by assigning task with great visibility into both our team and the whole organization. Task that might seem trivial such as participating in calls with his peer officers or running our meetings really built confidence and got me ready to take it to the next level. This is something I have been doing since day one I became a manager but I had never seen in practice. I got promoted to a management role like many others because of achievements and results in the operations area without little management training or experience and it was a little tough at the beginning, but while adjusting and keeping focus made a difference then, I can now appreciate how much better it could’ve been if we planned ahead. I also learned how to deal with a larger team and building a new team, not to mention how different it is to manage your team at your company compared to managing a team of volunteers.

I also learned that leadership is about value, communication and helping others grow. Bob did a great job helping me transition to my Officer role and I recognized that and will always try to apply it as a way to help other grow. As we were getting ready for full deployment of Win7 my director and other project leaders recognized that the 3^rd party Project Management that we had hired for the project would not be a good fit going forward. He did a great job getting us to a point but things started to not go so well during the first office deployment. My Director approached me and asked me to take over as the Lead PM because we had discussed moving PM under me based on success in projects that I had led. Because of my passion for this area I was excited and honored by just being asked to take over such an important task. However, I also recognize an opportunity to build and help someone grow. We had hired our current PM a few years ago as a PM, but he never really had a chance to lead any project because of many different reasons. I saw this as his chance to grow into the role and gain credibility. He had just completed his Project+ certification and I ask my boss if he would be willing to let him lead the project instead. He hesitated a bit but I asked him to trust me, and most importantly, to trust him and he agreed with the condition that I stayed on top of it behind the scenes, which I did by mainly emphasizing communications. The rest of the project was a big success and I was greatly satisfied by seeing this person succeed when people doubted him.

So what is it about value? This year, during some hard times when the dark side wanted to take over me, I really had to dig and rediscover my core values. I learned that if I stick to those my whole life will make more sense to me and my family anyway, and I will in turn, be able to add more value to those around me. I thank Randi Mayes, Executive Director of ILTA, for her wise words on this topic and continued commitment to helping ILTA's officers and volunteers to grow.

My most precious take away about leadership, is that I am now learning how to apply all of these concepts to my personal life. While I think that friends and family have always looked up to me and willingly followed me, I never saw it as an opportunity to lead. Leading my family is my most important task and growing together as successful people is my most rewarding and precious treasure and I can now do it with a leadership approach which is enhancing our already strong relationship and making our core values stronger. It doesn’t get any better than that!


That's it. It was a year of learning, and it will always be. I will soon post on what is ahead for the 2012 year. I hope that it bring energy, health and prosperity to all of you and your families.


Here to a great 2011 and an even better 2012! Happy New Year!

Win7/Office 2010 Deployment Office 2, Days 3 & 4

Last week I covered the first two days of our Window 7 & Office 2010 deployment in this blog. Today I am going to tell you how we close the deal during days 3 and 4.

Day 3. Closing the Deal

After a relaxing night closed out with a walk at one of Hilton Head beaches’ shore we were re-energized and it showed as soon as we got to the building. The first group, the support team, got to the office very early to be available to help the office staffers that got newly imaged PCs with the new OS and Office Suite as well as some new or upgraded applications, including our iManage DMS application. The deployment team arrived to the office later that morning to continue working on re-imaging machines and addressing escalated issues.

“Any change, even a change for the better,

 is always accompanied by drawbacks and discomforts.”

Arnold Bennett

Yes, there was a lot of change and the next few days, and weeks will bring a lot of adjustment for our firm while dealing with a new interface that touches pretty much everything they do in a daily basis. However, I can’t emphasize enough the great performance delivered by our Training Team. This showed during the whole day. While there were a lot of different activities going on at the same time, the day went very well; I really had expected more issues and questions from our users but they held up and dealt with the massive change extremely well and they actually did bring up some very valid questions. Once again, Kudos to our Training Team here, and they were not done yet.

After a nice group lunch in the office the big test for our team arrived; attorney training. The challenge wasn’t only to engage them and keep them involved but also efficiently deal with the many activities going on at the same time. We had two people doing floor support, two folks conducting attorney training, and three re-imaging the rest of the machines of those in training. Once I finished imaging one of the machines assigned to me I saw the need of switching my role into that of a Project Manager. Issues were still coming up and the vast majority was being efficiently and promptly handled by the support team, while some would need more advance troubleshooting and help from the engineers back at The Mothership. Thus, I saw the need to become the central information repository and I started to centrally compile all issues in our log dedicated to that. We always intended to approach the project this way but we had to deviate a bit from it for many reasons. However, we were able to quickly adapt and react to the new situation and get that PM role going again, which will now be part of all future role outs as it should. The PM will focus on being the “Central Communications Center” as well as the data collector so everyone can stay focused on their tasks. So I started walking around and making sure that everyone was doing what they were supposed to while collecting information that we needed to pass on to the rest of the team.

At the end, we were able to re-image all remaining machines and complete both support and training. It was a long day for most, but the preparation and planning that went on for months paid off again. I think that the main challenge for most was adjusting to the new iManage interface and Idol search engine as well as the obvious adjustment to the Office Ribbon.

Day 4. The Attorneys & Training; Need I Say More?

I do indeed! But this time I wasn’t impressed because of the amazing job that our trainers did. They really engaged the attorneys who also did a great job embracing the new interfaces and really going for it and trying to make the most out of the new systems.

“Change brings opportunity.”

Nido Qubein

As I mentioned these attorneys were really trying to adjust and make the most out of the system. They had great comments about the iManage EMM plugin for email management, which was not being used by all of them; some were exploring the IDOL search engine and began to like it a lot; heck we had attorneys using Win7 Snipping Tool!!! One of then used this tool to create a Memo and here is how she summarized her experience: “This upgrade has revolutionized my day! I did a two-page memo faster than I've ever been able to do before”. Now, getting this type of comment on Day One, I mean like the morning after the attorney was trained and facing massive change?…Priceless, for everything else, there are Windows XP & Office 2003!

Now, there were still issues and one that we are looking at how to better handle in the future is synchronizing OST files over the WAN. When a person travels to a remote office and logs for the first time to a different computer, Outlook will start synchronizing its caching and could potentially cripple the bandwidth to the office depending on the situation. The way we approached the initial deployment was by bringing OST file from last production day, in this case Friday, with us so that we could sync up locally without going over the wire. The switch to cache mode presents a new challenge not only during the migration but on an ongoing basis going forward, especially for offices with small bandwidth capacity. I will be looking at how our Riverbed Steelhead can help here.

Thanks to the amazing work of the support team, the deployment team was ready to return home by 3:00 pm. Our trainers remained behind and assumed support responsibilities the rest of the week which went very well. In fact, I’ve been checking with our service desk analysts and the call volume from the offices that have been converted have been very low according to them.

We are working on the next deployment which starts this upcoming weekend in our main office here in Columbia, SC. This will be the largest deployment so far and will tell us a lot for the rest of the way but I am confident that our team will deliver once again. From this point on I will be

occasionally posting updates about the whole project as we go on. I will continue to be in some of the deployments as either the Operations Manager or Project Manager. Meanwhile, my team is coming behind the Win7 team rolling out Endpoint and Media Encryption using Credant's Mobile Gurdian at the offices that have been migrated to the new OS already and I will be blogging about that project as well.

Win7/Office 2010 Deployment Office 2, Days 1 & 2

"Always plan ahead. It wasn't raining when Noah built The Ark"

Richar C. Cushing

So here we are, delivering Windows 7 and Office 2010 to Nexsen Pruet's second remote office in as many weeks. But first, a little background. First of all, I had not blogged about this project before because I have been in and out of it and my role has been changing. I have been primarily responsible for testing a few applications under my area to ensure functionality in the new environment. However, this week I am the on-site Operations Manager, which could translate into the on-site Project Manager/Do-What-Needs-To-Be-Done Guy (and everyone in the team really does what needs to be done anyway). I am responsible for making sure that the deployments are completed and that issues are addressed and compiled in a central place so that we can go over those before our next deployment. Finally, I'd like to add that this project started very early this year and there have been many phases. We started with brainstorming around January/February, to then go over a Branding phase in March/April. Around April, we began testing our many legal specific applications used by our Attorneys; to then begin creating and fine tuning images. It took until early August and 9-10 versions to get the right image. Developing the outstanding training material took a few months as well and delivering training started in July with Technology Department members first, and then to our pilot group. Last week we deployed the new desktop to our Myrtle Beach, SC office. This week is Hilton Head Island, SC's office turn. The best part of the project? EVERYONE in our department has had an impact, all of the eighteen people that conform our team, including temporary personnel has indeed contributed in a positive way, just as our very supportive and knowledgeable vendors have. But this is about the deployment phase, "The Real Deal". Let's take a look.

A team of six departed Sunday afternoon from Columbia, SC to our Hilton Head Island office. This team includes our Service Desk & Training Manager who oversees the delivery of training as well as floor support; the Operations Manager, who oversees the deployment of the images as well as the many applications and customizations required by each end-user as well as issues logging, one deployment engineer, one hardware and deployment technician and this time, our Director of Technology made the trip to compliment the team and contribute where needed. In addition, twelve others remain at our main office in Columbia holding the fort. Last but not least, I am proud to say that this is The Best Team and Technology Department I have been part of during my 12+ years career.

Day 1. Setup and Deployment

The team got in the office by 6:00 pm and by 8:00 pm the "Technology Operations Center", the Training Room and the temporary Video Conference room were all setup and tested. I have to admit that I thought that we had too many people, but boy was I glad we did! It did paid off. I was amazed of how well and quickly we worked together. This gave the training team the opportunity to go rest at the hotel to prepare for next day's class with staff.

Once everything was setup the deployment team started deployment of images to staff machines and we quickly ran into an issue. We are using Microsoft SCCM to deploy images and we are using our local File & Print servers as local Distribution Points to minimize WAN traffic. Well, the images did not finish deploying to the server. Now, I was out for a week at a Technology Conference and I kind of panicked for a minute and I think that so did our Director, but our Engineer who has done an incredible job on this project even well before we started, was prepared for it and had multiple DVDs with images for each type of machine at hand. I think that we may have lost perhaps a total of 15 minutes or so before we started the deployment. Our engineering and project management teams had prepared a couple of very detailed sheets, one with all the specifics about each machine that we had imaged with information such as specialty applications, printers, drive letters, and anything else that the end-user would need in order to complete their job; and the second sheet contains step-to-step details on how to deploy application and what to check off.

Imaging the machines may have taken about 30 min in average for each. Then we went on to deploy and test most of the applications in each workstation. I tell you, this is very tedious job and what took the longest because of how focused and detail you have to be in order to get everything done properly. We finish around 1:00 am, primarily because we were in such a roll that we just wanted to keep going. After all, we had the whole Monday to complete the deployment ahead of us.

Day 2. The Real Deal

Monday morning I met with the Training team and our Director joined us to go over what we had accomplished the night before and what was left to do before they head to the office to start training the staff. I followed the meeting with a 3 miles run around town, which was very helpful to help me prepare for the day.

The deployment team, including myself, got to the office later that morning while training was being conducted to finish the customization of the machines, which really wasn’t too much after all, the work the night before. Things such as printer drivers, and a couple of specialty apps were the only outstanding issues and they were completed by lunch time. I have to give credit to our Training Manager here; she did a magnificent job in terms of Coaching, Mentoring and Leadership while introducing this incredible amount of change to our users. Want to know why? This was our Trainer’s very first time teaching a Class in our Firm. We did hit a couple of issues but the planning and preparation that went on for months paid up during execution, a perfect example of the 80/20 rule, which if executed correctly does pay off big time.

The day could have not ended in a better way. Our seventh team member, one of our great Service Desk analyst arrived that night to do floor support along with the rest of the team and we head to dinner. After enjoying a great seafood meal at local restaurant, all seven of us took a walk at the beach, which was not only relaxing and needed but it kind of served as a team building experience. IT WAS AWESOME! We looked like seven children having fun!!

Meanwhile, back at The Mothership

Kuddos also to our EVERYONE back at our main office in Columbia. They were so responsive when we needed them and the communication across teams was incredible, another stepping stone for delivering this effort effectively. Our DMS, DBA, AD, Exchange and Network Engineers as well as our main Trainer, Service Desk and Project Management teams were always at hand when issues aroused and were able to quickly address them. In addition, they were also tasked to hold the fort and support day-to-day operations of the firm and they get as much credit as the onsite team does, especially being so shorthanded.

Days 3 & 4 include deployment and training for attorneys as well as support and I will tell you more about it at the end of the day tomorrow, but I can tell you that it was an exciting day for everyone.

Troubleshooting WAN Failover with BGP. A good procedure and attention to detail are critical.

A few weeks ago we ran into an issue when our primary Internet Service Provider (ISP) went down and we automatically failed over to our secondary ISP. This had happened before but in a different type of setup when we were in a dual homed Customer Router (CPE), which means that both carriers terminate in the same customer router; today both of our carriers terminate in their own CPE, which are Cisco Routers that terminate in a Cisco 3750 Layer 3 Switch Stack. The main points I want to make with this post are 1) you must have a structured procedure in order to troubleshoot this type of issues, and any other technology problems that you may encounter in your network for that matter. While it took a while to figure out what was wrong with this setup, I believe that it could’ve taken longer and I would have not been able to keep my cool had I not had a procedure to follow. And 2) pay attention to detail, it can save you sometime.

The problem we experience wasn’t really fail over. That did happen as designed. When our primary ISP went down BGP and EIGRP did all their work and the network connection came back up within 2-3 minutes after the failure. I followed our procedure to make sure that fail over happened properly; makes sure we are up (pings and alerting systems), make sure we know which network we are riding (trace routes and alerting system), user experience (make a couple of calls to make sure that systems were accessible), phone system is up (dial some extensions, check PRI registrations at the gateway). Up to this point everything seemed fine. I even called folks and they said everything seemed to be up. However, one person emailed me to report phone issues and at that moment I noticed that our PRIs were not properly registered with our Cisco Unified Communications Manager (CUCM). And here is where the troubleshooting began, not to mention that I am adding this step to the procedure, which is a “living” document.

I ran a few Show commands in both routers and the switch stack and all protocols were up as well as the main routes that we were riding; however, I noticed that my internal routes were not being advertised properly which led me to understand that our voice system was up by means of SRST, the fail over mechanism used by CUCM. I ran a few Show Run commands in all routers and switches in each side. I could tell that Router BGP and Router EIRGP had been told to advertise all the proper networks as shown below.

Remote Switch                                                                                                               Central Location Switch

                    

I then turned my attention to the BGP routers and again, everything looked fine there as well.   

Central Location (Hub) Router                                                                              Remote Router 

I also did run a Show IP Route as well as Show BGP in the routers and Show EIGRP as shown below. I realized that the internal networks from my remote office were not being advertised back to the Hub site but I missed what the main problem was, which I will explain shortly (screenshots below are from current config and they don’t reflect what happened then. Our 10.70.0.0 network was not showing at the moment)

 

Show BGP told me that I was missing remote LAN network (10.70.0.0 was not showing up)

Some EIGRP Stats

The next step in my troubleshooting procedure was to contact our primary ISP to confirm whether or not they could see the routes that we were sending through our secondary provider, and sure enough, they could not see them. At this point I really felt lost and followed the playbook which says: “Stop wasting time and call Cisco TAC”, so I did. After about 20 minutes of troubleshooting (that after the frustrating 30 minutes on hold), they finally identified what we were missing. CHECK YOU ROUTER ID (RID). The engineer realized that the BGP RID in both routers were the same, hence EIGRP was not able to send the routes properly because it had two BGP routers with the same ID. Once we changed the RID in one of the routers (in our case the primary router) the routes started to propagate accordingly and we were 100% operational while in failover mode. To change the RID go into Router BGP mode and then run the change RID command, where A.B.C.D is the IP address of the interface that you want to designate as the RID.

I am glad that we worked on this until we resolved the issue because this was a very long outage and we were still riding the alternate ISP the next morning. Bottom line: have strong troubleshooting procedures and methods, and revise them as your network changes and evolve, understand what the problem is so that you can tackle it accordingly and pay attention to detail. It can save you time and many frustrations. In addition, I checked other remote sites and noticed the same problem, which I addressed and now all branches are setup properly, which will eliminate further issues.

Using Managed Services and SaaS in your Information Security Strategy Makes Sense.

Over the last several weeks we’ve seen how big corporations have been hit by security incidents. Those who made the news were often because of incidents related to Data Loss, such as WikiLeaks or Episilon events. They show the need for security professionals in the enterprise is increasing. And there are other areas of concern like the current landscape of advance threats, internal threats from disgruntled employees or insider trading, or increase in usage of services like Dropbox, Skype, and mobile devices among others. And when solid Information Security vendors such as RSA and Ashampoo experience data security breaches, I just wonder if there is hope for the rest of us? I believe there is and it may not be inside your organization.

When it comes to information security I believe that you must hire the best resources that you can. Whether internal or outsourced this group must have the skills, knowledge and access to do what they need in order to preserve your company’s data, which should be one of the most valuable assets of the organization. And hiring partners in the form of SaaS and Managed Service Providers, MSP, makes sense because hiring your own resources with those skills will certenly be very costly.

We are a small team of two and half that is responsible for Network Infrastructure and Security for an eight offices, 400 employees law firm. None of us is a full time Security Professional dedicated to the area and that is why I have been making strategic alliances with our Security Vendors that can help us built a strong security team. For years we have partnered with DELL SecureWorks, one of the world’s strongest Security MSPs. Although all of us are very good security engineers with security certifications and strong skills, we just don’t have the manpower to dedicate a FTE to be watching firewall logs and alert us of possible incidents. The superior work that SecureWorks does led me to grow our relationship by also outsourcing two areas that are important to any security program, especially when regulations like HIPAA for example, are part of the conversation, such as IDS/IPS and Log Retention. They add value to our team by tackling the biggest challenges that these technologies present: keeping up with the logs and alerting when suspicious behavior is present. We still have to do our part, which is reacting to the alerts and mitigating the threat, but we have been able to react and pull machines out of the network or close a whole within minutes. Even if we had a dedicated resource for this area, I don’t think that we would be able to react and take action that quickly.

We’ve also hire SaaS companies to help us secure other areas of our network perimeter, specifically email spam, malware, and DLP filtering, and also Web Content Filter. We are currently transitioning our email edge security to Proofpoint, which has immediately added value to our security program with its very strong DLP engine. There are two things I like in particular about this vendor; one is that we don’t have to maintain the DLP dictionaries, something that most vendors would defer to you. The other one is that the appliances which are not in our premises, would attempt to make a TLS connection to the peer email server(s) and if it can’t and there is sensitive information, then it would send a secure message to the recipient. We still need some folks to look through logs and take some actions but less is required from our team. This setup is becoming kind of the standard on today’s email security practice.

At the web browsing edge we have merged from a complex in-house solution composed of three different vendors to another SaaS solution with ZScaler. Since merging to it, the Malware infection in our machines has decreased by 60%. We filter through many gateways in Zscaler’s private cloud by putting a PAC file in the machine’s web browser and the user then filters through the closest gateway to her. This is generally kind of pre-set when the user is in the office because we would always hit the closet gateway to our data center, and will fail over to the next closest one if the one goes down. Now we also have the ability to protect our laptop users when they are outside our offices. In that case, the Zscaler’s Geo-Location feature kicks in and the user browses through the closest node to her, whether she’s in her house, California, or Europe she will always hit the closest gateway available and proper security policies will be applied. The only time when we get involved is when a website is blocked and it someone needs access for business reasons. Many other capabilities are available with this engine, such as throttling bandwidth for media streaming or file transfer, which we use, DLP, which we are testing, or ability to prevent users from posting to media sites such as Facebook or Twitter. It can also block web access from pre-determined browsers, such as old IE, or Firefox for example.

Our team is still responsible for managing areas such as Anti-Virus and Malware, securing network gear, Server and Workstation patching, some areas of physical security and soon HDD and Media encryption, which are all candidates for outsourcing as well. However, we are much more effective by working with trusted MSPs and Security SaaS vendors than if we did it all in-house because. First of all, proper staffing to achieve the same goals would be costly and today is simply out of reach, and second, they can help us keep up with the ever changing and developing threat landscape while reacting to real attacks in a much faster and effective way. In addition, I can now concentrate in developing Policies and Procedures, Incident Response, as well as the other operation areas that me and my team are responsible for. It just makes sense to go this route as opposed to investing on in-house skills.

ILTA Conference 2011. Server Ops and Security, and Tech Ops Teams will deliver great content

2011 rev-elation Conference is approaching and our Server Ops and Security PG is bringing great educational content to it. Our fabulous Steering Committee team, led by Bob DuBois, and completed by myself, Tom Crowe, Mark Brophy, Nate Smith, Dave Nevala, and Toni Brester has done a great job developing sessions around topics often seen in the discussion boards around different technologies that support our demanding law firms. But first, a quick overview of the Conference Team.

The journey began on Friday August 27^th of 2010 while many of us where returning from the fantastic 2010 Strategic Unity in Las Vegas. Right then, our awesome Co-Chairs two-year veteran, Meredith Williams, first-year Co-Chair Kathy Lentini, BoD Liaison, Eric Anderson, along with ILTA’s TJ Johnson and Peggy Wechsler, began looking at their strategy to tackle this year’s conference. As for myself, I am on my 2^nd year representing the SOSPG in the Technology Operations Track. Meredith and Kathy made a great choice by appointing Skip Lohmeyer as Tech Ops Team Leader and together they brought in an impressive group of Legal Technology Professionals that make up our team. The same is true for the other three teams that complete the whole Conference Committee, which are Information Management, Organization Management and Applications/Desktop. Together the committee has come up with almost 200 educational sessions. I thought that last year’s conference was the best that I had ever attended; today I realize that I will say that after each conference that I attend. It is just amazing how the team came up with this amazing content, and how ILTA does it every year. Last but not least, Gaylord Opryland Resort has made an impressive recovery work after being 8-10 ft. under water after the flood in Nashville last year and it is looking better than ever; what a gorgeous venue for our Conference. Oh, and they will tell us their recovery story in Conference. You can’t miss it.

I can only give you a 10K ft. view of what we are bringing to conference as a Peer Group but I am very excited about our sessions and I know you will as well. The team tried to create good balance to bring relevant content that can have an immediate impact on technologies that you are currently evaluating from a servers operations and security perspective. In addition, the whole Tech Ops Team is delivering equally well balanced and amazing advanced sessions in areas of technology infrastructure that support our computing networks. Here is a quick view:

IPv6. Yes is finally coming. And it will be at ILTA this year as well. We are teaming up with the Emerging Technologies PG to deliver two sessions that will unveil IPv6 for us and what it means to how will be running our networks soon. As a reminder, watch out for “World IPv6 Day “on June 8^th. If you thought this is not relevant think again as Asia have ran out of IPv4 and Europe is next as reported by NetworkWorld here http://tinyurl.com/65e6d7v

Endpoint and Media Encryption. Three different Law Firms will tell their story on why this technology is needed, what’s available, what to watch for. There will be content for all firm sizes here.

VDI. We’ve done this one before, but never at this scale. See how two very large firms with global foot prints have delivered hundreds of Virtual Desktops effectively. An impressive session is being developed with equally impressive speakers.

Securing the Virtual Environment. You have to come see this one because traditional security no longer applies to the virtualized world.

Upgrading to Exchange 2010. How do you prepare for it? What challenges can you encounter when coming from different previous versions? Two firms and one of our great ILTA Sponsors will prepare you for it.

Technology Operations Forecast. The way we design, implement, and manage technology will not be the same. This session is part of ILTA’s Law 2020 initiative.

And there are other amazing sessions being delivered by the Tech Ops Team which include, a Disaster Recovery session where the Gaylord will tell their story after the flooding and a global firm with offices in Japan will do the same after the massive earthquake and Tsunami that hit them this year. Other Sessions include DLP, Exploring the Exchange 2010 Ecosystem, Hosted Communications (both voice and email), Securing Windows 7, Keeping Documents Secured in Mobile Devices, and Change Management Impact on Tech Ops.

Go register for this year conference, and when you get there, check out the SOSPG group and Tech Ops sessions. You don’t want to miss this year’s conference! And please join me on thanking the amazing people behind our PG and Conference Committee teams for contributing with their time and knowledge to help ILTA make this great conference happen!

About ILTA. For over three decades, the International Legal Technology Association has led the way in sharing knowledge and experience for those faced with challenges in their firms and legal departments. Through delivery of educational content and peer-networking opportunities, we provide members information resources in order to make technology work for the legal profession. Visit ILTA at www.iltanet.org

I Shall Become a Virtual Machine

“As far as personalities are concerned, if men are from Mars and women are from Venus, then IT people are from Microsoft and their business partners are from Apple.” Susan Cram in her book 8 Things We Hate About I.T.

The statement above got me really thinking, observing, analyzing, and connecting. At the end I concluded that I wanted to become a Virtual Machine. Let’s take a look.

IT people are generally seem, rightfully so, as Geeks and according to Paul Glenn from Leading Geeks they “are more captivated by technique than application.” This next statement in the book made me think about the Apple vs. PC commercials. On the one hand we have the Geek representing PC, or really Microsoft, who needs to do so much work to get a task done. He also struggles explaining the How, but cares little about the Why. On the other hand, we have the modern guy who just gets the task done. His main concern is to efficiently do what is needed from him while also being innovative and creative.

The problem is that they constantly fail to communicate, at least effectively, which is one of the big problems between IT and business units. Business Leaders are concern with strategy, growth, revenue, profits. And they should. However, more often than not they assume that IT can deliver what they need, the way the need it, and when they need it without discussing the current business goals and strategy with IT Leaders. On the other hand, IT believes that they know what they are doing and that they are delivering value to the business. Many times we assume that we are valuable to the business because we just delivered the newest, fastest and coolest Intranet, Phone, Email, Billing or ERP Systems. But is that what the business need in order to growth, generate revenue, and profit?

Connecting Apple and Microsoft

Then I started thinking about Citrix and VMware, two major technology players today that deliver Virtualized platforms such as servers, applications and desktops. Why? I own an iPad and I absolutely love it. While I don’t consider it my primary tool to conduct my day-to-day work (yet), more and more often I use the Citrix Receiver on it to connect to my Firm’s system and do my work. Others may use VMware View client to get the same work done, and again, the common theme between those two is that they are virtual platforms that are pretty much hardware or OS agnostic. Their sole focus is to Connect you to your office so that you can do your work. While I understand that this approach may still not address the two platforms failure to communicate to each other, I do believe that it addresses the disconnection between them because I can now run a Microsoft Desktop or Application, in an Apple device (chuckling while picturing Steve Jobs laughing about it)

Becoming a Virtual Machine. Becoming a Connector

And that is why I want to become a Virtual Machine (VM). I want to be able to connect Business and IT by establishing and maintaining an effective communication bridge between the two so that IT can deliver value and operational efficiency. I want to be able to help the whole IT Organization to understand What the business is trying to achieve, Why is trying to achieve it, so that we as an organization can be part of the “How we can achieve it” discussion. I want to help the Business Units to understand what the current capabilities of the IT Organization are, and what they might be in the future. I want to be able to connect both sides to deliver services that will help us become more operationally efficient as a whole, so that Technology can be used to create innovation and competitive advantage.

So I guess I want to be a VM, and next time you see me you may call me VM.

And I leave you with a question. Who is Google then? I believe that it is the Staff in general. The Staff outside Management or Leadership of any sort, which despite getting it done is having a hard time being considered into strategic discussions, but it is kind of making its way through, at least in the most efficient organizations. Food for thought.

The Impact of Change Management on Technology Operations

For the last 18 months or so I have been researching and advocating for the need of having a better control of change in our environment. I strongly believe that Change Management is not “a thing for developers and project managers”.  It is widely known that many times we cause our own when we upgrade a system, or make a small change to a configuration, that triggers, sometimes not immediately, a sequence of events that can have a negative impact in our network operations and services that we provide. We have implemented a series of tools that have started to have positive impact in our Technology Operations.

Our Tools

The first tool that we introduced was DeviceExpert from ManageEngine, which added serve as our Backup tool for Network Gear and had a Change Management module that soon became popular. Because of this software, I started to understand how Change Management could help us improve troubleshooting, and operations of our network, and I started “evangelizing” about the need for a tool that took it beyond that area and covered as many systems and processes as possible

About six months our Technology Director and I started discussions around how it could help us and we soon added the rest of our Management team and we were all on board. Armed with all the input that we gave him, our Director designed and developed a Change Module that was added to our Resource Management and Systems Catalog portal which we named Techsystems. We use this module to log changes that we make to the systems and processes that we support. Granted, it is a manual process today, but it is already helping us resolve operation issues and it has great potential.

So How Does Change Management Impacts Operations?

In my opinion, one must have feature of a tool like this is the ability to create and schedule reports, and actually look at them. Both of our systems deliver reports that we review individually and then as a team during our weekly meeting. The result? Better picture of what’s happening and possible impact of changes made that were otherwise previously unknown.

I’ll now give you a couple of practical examples of how it helped our team overcome a couple of issues. One day we started receiving blank faxes from our Fax Server, which usually means that there is a communication issue somewhere.  I knew it was a missing setting in either our Cisco Unified Communications Manager or one of our routers that server a gateway to the server, but I couldn’t remember where it was. We spent the next couple of hours going over different settings in the Communications Manager and some of the routers until I finally remember that there was DeviceExpert and shouted it out! We immediately turned to it and ran a Change Management report, which showed the setting that was missing in one of the Routers. Issue resolved. Our problem was not looking at this report earlier because we had never used it. Since then, I have made this my first step to troubleshooting where it fits and I encourage our engineers to do the same. That’s where the value of the tool is. It’s just not a log that you never look at; it is a tool that can help that should be used.

Last week our Financial Systems Administrator found out that our calls had not been properly billed since January 11^th. Not good in a Law Firm. He spent a couple of weeks trying to figure out what the problem was because other members of the team had been busy traveling or in projects. This past Monday we finally had time to talk about it and my first questions were to describe the situation and tell me when this started happening. With that information we went straight to Change Management and looked up the log to see what had happened around that time, and sure enough, there were some changes made in the server that host this process. Then, we were able to trace the problem back and revert to original configuration and in about 20 minutes the process was working properly.

If you change it, log it. If you log it,  look at it and share it. What are you doing that is helping you with Change Management?

Launching the Hands-on Technology Operations Manager Blog

Thank you for visiting my blog. My name is Carlos Rodriguez and I am very excited about my new blog. I will be blogging about topics such life of a hands-on Technology Managers, various technical subjects that I deal with in day-to-day operations of the network that I oversee, projects that I am working on with my team, and sometimes personal topics as well.

Who am I?

I’d like to tell you a little bit about me. I am originally from Venezuela and moved to The US in 1999. I have been married to my incredible wife Johanna for 11 years now and live in Columbia, SC where we settled since we moved in. We have two children.

My Experience

My career in the technology world began in 1997 in Venezuela while I was still attending school.  I obtained my BS in Computer Science degree there in 1998. I started as a Field Technician and then went on to do some Computer and Networking Sales and Consulting on my own. I then moved to The US in 1999 where I have worked in technology in industries such as Call Centers, Consulting and Legal. Throughout my career I have work as Hardware Technician, Data Processor, Systems Administrator, Help Desk Technician, and Supervisor, Network, Voice and Security Engineer. Today I work for Nexsen Pruet, LLC Law Firm, where I started as a Systems and Voice Administrator and now oversee the Network Operation of a mult-isite organization running on MPLS as the Network Manager, where I am responsible for the team that oversees the day-to-day operations in the areas of WAN, LAN, Voice, Data Center Infrastructure and Network Security, while assisting in areas of Mobility, Citrix, Servers and Virtualization.

Beyond Technology

I constantly seek personal and professional development through reading, and training. My core technology areas are Networking, Voice, Information and Network Security, and Server Operations; which has led me to seek and achieve certifications such as CCNA, CCNP, CCVP, MCSA and Security+. In addition, I will be pursuing other certifications related more to Technology Management such as Project Management Professional, and ITIL. I am very passionate about constant improvement. I believe that my mind is the best tool at my disposal and that I can create my own future through, therefore, I train it just as I train my body and my technical skills.

Leadership: My Passion

In addition, I serve as a volunteer in several organizations, including The International Legal Technology Association, ILTA, The Carolinas Cisco Users Group, CCUG and The Information Technology Council of the Greater Columbia Chamber of Commerce, ITC. I am a contributor in those organizations in areas of Leadership, Authoring having had a few articles published, and as a Speaker.

My goal with this blog is to share my experiences managing a team and network in a medium size environment, as well as creating a forum where we can discuss ideas and share practical examples so that those of us in similar environments can learn from each other and apply them in our environment when possible. I truly believe that I am not alone, and most importantly, that we can no longer isolate ourselves from those that share the same challenges. Collaboration is one of the keys to succeed in what we do. In addition, you will see a lot of posts related to the Technology Operations within the Legal Industry, which is my the vertical I work in.