Wednesday 12 November, 2008 (.Net | BizTalk | Ramblings | Team System | Tech Ed 2008 | Visual Studio)
This is my third day at Tech Ed 2008 in Barcelona, the breadth of the technologies and languages that we as .Net developers now need to at least understand and preferably master to some degree is enormous. I believe that developers will need to specialize more explicitly on certain technology areas to remain productive in the future. It has been impossible to master all of Microsoft's products for many years now but up until a year or so ago it was at least conceivable for a developer to master all of the developer technologies that Microsoft had on offer. With the introduction of cloud based services and all the news in .Net, both libraries and new languages I cannot see how any one person will be able to master it all... I want to sum of some of my thoughts on the sessions that I have attended here at Tech Ed on Visual Studio 2010.
Fixing Continuous Integration
CI is a great principle and has moved into the mainstream along with many other agile principles. While Continuous Integration as such has never been broken there has always been a problem with developer discipline. If you check in something that is broken then it will be discovered quickly with CI but if you don't fix the broken code other developers will be less productive if they cannot get the latest code or have their own checkins fail in CI due to previously broken builds. With Visual Studio Team System 2010 Microsoft is introducing a new concept for CI, the gated check. It is not a big change to the CI flow but a very clever use of a shelving, a feature in TFS which I find generally is underutilized. Instead of doing a standard checking in and then running a build on that code, gated checkins shelve the developers changes and then performs a build on the shelveset. If this build is successful then the shelveset is promoted to a proper check in, if the build fails then the check in is not committed to the mainline. The visibility that CI offers is maintained but using gated checkins other developers will never get failing code from other developers which was discoverable! Obviously undiscovered errors can still sneak into the code due to bad or missing tests or incorrect configuration of the build environment but it I believe that this will lower or eliminate the fear of checking in that some developers have. It will also allow you to check in before you go home without the need to wait and see that the CI build completes successfully. If I check in my code 5 minutes before I go home I should ideally wait to see that the build is successful, if it does not then I need to fix that immediately since other developers will be getting that code tomorrow morning when they come in. Using a gated check in I can go home, let the CI build complete on my shelved changes and if the build is successful then everyone can get my changes in the morning, if it fails I can safely wait until the morning before fixing the code and no one else is impacted by my errors.
Reproduction of Errors and Architecture Compilation Failures
Visual Studio 2010 looks like it contains some pretty good testing tools like the ability to record a testers GUI while performing manual tests and also including debugging and tracing info from remote machines. Pretty powerful stuff! It seems Microsoft have their mind set on eliminating the problem of irreproducible errors.
Another interesting feature is the architecture layer diagram which allows you to specify how the different layers in your application may and may not call each other. This is useful for validating that your code follows the intended architecture at compile time instead of detecting it when problems arise. All of these features (and there are more) are great but unfortunately Microsoft still has silly distinctions between different editions of Visual Studio Team System, so many of the cool demo features will probably be marginalized in most organisations due to them not actually having the appropriate edition of Visual Studio.
Monday 03 November, 2008 (.Net | BizTalk | Fixes | Java)
Working with Websphere MQ and the MQSC adapter is quite new for me and I have been struggling with a couple of problems which have been very difficult to find solutions to. One of our main issues has been getting the correct character encoding of outgoing messages to MQ. BizTalk would output the message with the correct encoding (using a file send port to verify) and it looked like the messages on the WebSphere MQ queue were correctly encoded when we checked them using rhutilc. In our case we encoded our unicode messages as UTF-8 but when the Java system which is the destination read the messages any non-standard US characters came out as rubbish. They were correctly encoded as two bytes in the message but the destination system was reading them using codepage 850. Also all unicode messages that the Java system sent to us encoded with UTF-8 over MQ worked correctly in BizTalk.
What I have found out through much trial and error and the sparse clues I have been able to find on the Internet is that the WebSphere client that BizTalk uses under the covers will set the character encoding of any messages that you write to the systems default codepage, which for windows is 850 according to IBM. This setting does not take into account the actual encoding that BizTalk 2006 has done to the message. If you force your message to be encoded as 850 in BizTalk then it will most likely work as long as you don't use characters that are outside the codepage. There is a context property in MQSeries.dll called MQMD_CodedCharSetId which will allow you to set the codepage value (CCSID in the MQ world) for your outgoing message.
I have found three ways to remedy the encoding problem:
- Set the context property in an orchestration to whatever value you require. The drawback with this is that if you need to change your encoding you have to recompile and redeploy everything. Also changing your context property does not actually change the encoding just how MQ should interpret the message.
- Set an MQ environment variable named MQCCSID to the codepage that you send your messages as. This will override the system default value and force messages to be tagged with your required codepage. Obviously you can only have one system wide default so if you need to send messages out with different encodings this isn't going to help. Again the actual encoding of your messages is not going to be changed.
- Create an encoding pipeline component that allows you to set the MQ codepage for an outgoing message. This is the option that we have settled on for now. The encoding of your message is still not actually changed, unless you implement it, but now you can have different encodings for different destinations and you can change the encoding value using MMC and binding files.
One important point to remember is that UTF-8 is codepage 65001 in Windows but 1208 in IBM, so wherever you set the codepage you need to have this in mind - as far as I know all other codepages have the same ids.
Using rfhutilc you can verify the codepage of a message in an MQ queue by browsing to the message and then checking what codepage has been set under the MQMD tab.
I have not verified this but the solutions that I have outlined above should work for EBCDIC and any other odd encodings that you may require as long as you can actually create a message with that encoding.
Friday 10 October, 2008 (.Net | BizTalk | Fixes | Tips)
The BOM is a two to four byte sequence at the start of an encoded file which tells the recipient if the characters are encoded using big-endian or little-endian byte order. In BizTalk generally the byte order mark is added when a file is UTF-8 or UTF-16 encoded using an XML assembler in a send pipeline. Sometimes applications cannot handle the initial bytes and you don't want them to be added to your message. I recently read two articles about removing the Byte Order Mark (BOM) from outgoing messages in BizTalk 2006 one by Saravana Kumar and the other an official Microsoft KB article. Both of them recommend creating a custom pipeline with an XML assemble stage where the PreserveBom property is set to false. I find it a little strange that they don't point out that you can accomplish this using the default XMLTransmit pipeline also. Just open the pipeline configuration in the Send Port properties and set PreserveBom to false... No custom Pipeline, no coding required.
Wednesday 08 October, 2008 (.Net | Bugs | Fixes | Vista | Web)
While trying out a complex URL rewriting scenario in ASP.Net I ran into a pretty strange problem that resulted in a 404 error:
Description: HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable. Please review the following URL and make sure that it is spelled correctly.
Requested Url: /6.aspx/t.aspx
My code is very similar to a solution that I have created previously in ASP.Net 2.0 on Windows XP so I was a little surprised that I was getting an error. The strange thing is that /6.aspx is the original URL and /default.aspx is the rewritten url, in the error message it looks like the original URL has partially overwritten the new URL!
The exception that is caught has a stack trace that ends with: System.Web.UI.Util.CheckVirtualFileExists(VirtualPath virtualPath)
Looking around the Internet for people who have already encountered this problem (as I always do) didn't lead to any immediate solutions. On the IIS forum there is one discussion on Module doesn't work on IIS7 under vista business which lead me to a solution. It seems to me that URL rewriting is broken on Vista prior to SP1. Part of the SP1 upgrade incorporates changes to IIS 7 that were made in Windows Server 2008 as detailed on Microsoft TechNet. So after installing SP1 my solution works as expected.
Thursday 26 June, 2008 (.Net | Fixes | Ramblings | Web)
I was just surfing around on http://forums.microsoft.com and got a yellow screen of death!
I must say I was slightly surprised. In web projects that I have been involved in we have always added two customErrors pages one to handle generic errors which redirects to an errorpage.aspx page and then a static html page errorpage.html which is only used when an error occurs on the dynamic error page... A sample configuration using location follows:
<configuration>
<system.web>
<customErrors mode="on" defaultRedirect="errorpage.aspx"/>
</system.web>
<location path="errorpage.aspx">
<system.web>
<customErrors mode="on" defaultRedirect="errorpage.html"/>
</system.web>
</location>
</configuration>
That way if we have a problem with our application that affects the entire site, for instance menu and navigational rendering, we can still show a nice error page with a message like "The site is temporarily unavailable, please try again in a few minutes."
Thursday 26 June, 2008 (.Net | Bugs | Fixes | Team System)
We have just had a major Active Directory issue that has affected the Team Foundation Server instance that all the developers at the client that I am working at use. The problem occured when some users where erased in the Active Directory and were subsequently recreated with the same user names. These new user accounts cannot access the Team Foundation Server.
A large part of the problem was solved by just removing users from TFS and then putting them back in again. This recreates all the wiring under the surface in TFS so that the users can access the server but all their existing workspace info, pending checkins and so on are lost locally yet exist in Team Foundation Server with a different Owner under the same user name... The only way to tell them apart is by their SID and that the original account may get suffixed with a number in workspace lists: AD\username:61.
My first thought was to remove all the workspaces and pending checkins and let people manually resynch their projects. When I clicked undo pending change for a bunch of changes in Attrice Sidekicks an unhandled exception popped up: "Object reference not set to an instance of an object". I just can't undo these changes, there is a clue to the problem when I try to unlock a changed file, I get an Unlock error:
Failed to unlock $/Project/Path/file (TF14061: The workspace COMPUTERNAME; AD\username does not exist.)
I can't search for workspaces by username either, even doing a simple TFSSecurtiy /i "ad\username" /server:TFSName for one of the affected accounts returns "Error: The identity cannot be resolved." Since the original accounts have been removed from Active Directory I cannot easily get hold of their SIDs for querying with TFSSecurity. Attempting to delete the workspaces from the command line return the following error:
TF14061: The workspace WORKSPACENAME;AD\username does not exist.
Things seemed pretty messed up... I found some clues as to what was required to get things working again:
http://support.microsoft.com/kb/948679
http://support.microsoft.com/kb/823278
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=454060&SiteID=1
According to the KnowledgeBase articles trying to create a new project while TFS is in this state may result in the following error:
TF30170: the plugin Microsoft.ProjectCreationWizard.WorkItemTracking Failed during task WITS from group WorkItemTracking
Basically Team Foundation Server has a bug that causes deleted accounts to not be resynched correctly and SharePoint doesn't handle recreated user accounts at all - you have to resynch them manually. Unfortunately in addition to this there is a bug in TFS SP1 which prevents TfsAdminUtilI from correcting the SIDS! You get the following error:
ERROR: Could not access database.
In the end I managed to correct all the problems, here are the steps required:
- Contact Microsoft Support and request the hotfix for http://support.microsoft.com/kb/934216
- Install the hotfix...
- Follow the steps in http://support.microsoft.com/kb/948679 to resynch the users SIDs in TFS
- Follow the steps in http://support.microsoft.com/kb/823278 to resynch the users in SharePoint
- Recycle the TFS App Pool to force an update of all users and groups in TFS.
Look out with users who have their workspaces mapped to "c:\documents and settings\" the new user accounts cannot necessarily access their old files there since they now log on with a new account (which just happens to have the same name).
Thursday 15 May, 2008 (.Net | Bugs | Java | Web)
I have been having some trouble lately trying to debug a .Net web service that I have constructed for a proof of concept. The scenario is that we have a Java application that calls a web service and the web service passes the call on to BizTalk for processing. Not a terribly complex scenario so I wasn't expecting much trouble setting up the POC. Unfortunately we have experienced some really strange problems. Calling the web service from a .Net client works without problems but when the exact same SoapEnvelope is used in the Java client through a proxy we get a "HTTP/1.1 400 Bad Request" error!
How is this possible? Well after much trial and error, searching the net and generally scratching my head I finally found a clue to the root cause of the error. One of the scenarios when "400 Bad Request" is returned is if the value of the content-length header is smaller then the actual length of the message body. I could not find a simple way to get IIS 6 to log the actual HTTP headers which were received and my favourite HTTP debugger Fiddler couldn't be used when the calls came from the Java client so it was very difficult to verify. The WSE input tracing showed a correct request and the application log contained the following unhelpful message
Process information:
Process ID: 5856
Process name: w3wp.exe
Account name: AD\serviceaccount
Exception information:
Exception type: HttpException
Exception message: Server cannot clear headers after HTTP headers have been sent.
I managed to get a nice raw HTTP dump sent to the application log by adding the following to my Global.asax
void Application_BeginRequest(object sender, EventArgs e)
{
//save position for reset
long position = HttpContext.Current.Request.InputStream.Position;
string requestBody = new StreamReader(HttpContext.Current.Request.InputStream).ReadToEnd();
//reset the position
HttpContext.Current.Request.InputStream.Position = position;
EventLog.WriteEntry("Service HTTP Dump", Server.UrlDecode(Context.Request.Headers.ToString()).Replace("&", Environment.NewLine) + Environment.NewLine + requestBody);
}Using this I could verify that it was in fact an incorrect content-length in the HTTP header which was causing the problem. Using Notepad++ and Fiddler I could see that the actual content-length was 634 using CRLF and 631 using only CR. The Java client was specifying 631 but the .Net web service was actually receiving 634. With the help of Telnet I could send in a call with 631 characters and CR as line-end and verify that the .Net web service handled the request properly. For the sake of clarity this is a sample Java HTTP header:
POST /path/service.asmx HTTP/1.1
Host: servername:80
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Content-Type: text/xml; charset=utf-8
Accept: application/soap+xml, application/dime, multipart/related, text/*
User-Agent: Axis/1.1
SOAPAction: "http://schemas.example.com/2008/04/22/Service:ValidateRequest"
Content-Length: 631
The same request from a .Net client:
POST /path/service.asmx HTTP/1.1
Host: servername:80
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.1433)
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://schemas.example.com/2008/04/22/Service:ValidateRequest"
Content-Length: 634
Expect: 100-continue
Both requests had a length of 634.
More info on the exact nature and calculation of the HTTP headers are in the RFC and in section 3.7.1 of the RFC valid line endings for the message body are defined. Basically it says that CR, LF or CRLF should be valid as long as they are used consistently but CRLF should still be treated as two characters for content-length (the spec doesn't explicitly say that but I presume that it is required). So the .Net web service has a correct behaviour, the specification also says that content-length may be omitted in HTTP 1.1 although it should be honoured if it is specified. Since it is required in 1.0 and it is usually very easy to calculate it isn't something that you usually have to think about trying to omit and it is hidden in the web calling infrastructure of Java and .Net.
I am not sure where our error is introduced. It could be when the original message is constructed in Java, whatever library is used to calculate the content-length could do so incorrectly (not likely). A more probable source is that the message is constructed with correct line-ends in the body (CR or LF) but that the intermediary proxy recodes the call with new line-ends (CRLF) without updating the content-length.
How can we fix it? I believe that there are three possible solutions for the problem (we haven't solved it yet).
- Remove the proxy, should be the easiest but due to the network configuration that isn't possible at the moment.
- Change the way that the Java client creates it's content so that it uses CRLF by default thereby forcing a correct larger content-length.
- Use the Application_BeginRequest method in Global.asax or an HTTPHandler to somehow rewrite the input stream or recalculate the content-length. At the moment I am not sure that either is possible.
Update
We managed to remove the proxy (which was actually part of a VPN connection) and our troubles disappeared.
Tuesday 15 April, 2008 (.Net | Blogging | Bugs | Fixes | Web)
I seem to have managed to get server pings from dasBlog to work in medium trust! This has been a major issue for me previously, not as much of an issue now that I primarily use off-line tools but it is still great to have it. The key is that originUrl in the medium trust settings supports regular expressions! More info on Haacked, which is where I found the solution. Just to make things clear:
<trust level="Medium" originUrl=".*" />
will allow your web site to send web requests to any external site...
Tuesday 15 April, 2008 (.Net | Blogging | dasBlog)
It turns out that it really was very simple to upgrade to dasBlog 2.0 and ASP.Net 2.0. I did the ASP.Net upgrade first simply by changing which version of .Net to use. dasBlog 1.9 seemed to keep on working without a hitch which meant I could proceed with the upgrade process locally. I followed the instructions and uploaded all the files to my server...
Unfortunately I had missed a section that I had changed in my web.config which of course meant an ugly error page was the first thing that came up. In the upgrade instructions they point out that the most common cause of errors during an upgrade is problems merging changes in web.config so I could easily fix it guided by the detailed error message that I received.
Now it seems that most of the problems that I had with previous versions of dasBlog running under medium trust are gone. The only problem that remains is that the default configuration of medium trust won't allow dasBlog to ping or trackback to other sites, I can live with that!
Tuesday 15 April, 2008 (.Net | Blogging | dasBlog)
I am preparing to upgrade my blog to 2.0. Thats both dasBlog 2.0 and .Net 2.0.
It looks like this version should fix all of my previous issues with my setup, unless my host has done some weird .Net 2.0 medium trust setup, you never know... There is a guide on doing the upgrade install here. As far as I can see the process seems to be:
- Download & place all the files for you new version of dasBlog locally
- Download the content folder from your public site and place it in the local version of dasBlog
- Run the upgrader tool locally
- Merge changes in your current web.config and the new versions web.config
- Merge changes in your current site.config and the new versions site.config
- Upload all the local files overwriting existing files
Seems pretty simple. I also have to change the config of my host to use ASP.NET 2.0.
What could possibly go wrong... 
Monday 04 February, 2008 (.Net | Bugs | Fixes | Web)
Recently I was working on ASP.Net 2.0 Web Parts in a Virtual PC environment and ran across a strange exception. I had read that the Web Parts personalization database is separate from the membership and profiles databases but had not really thought about it further. Previously everything just seemed to work once I ran "aspnet_regsql" to create the aspnetdb database to handle all the data for the services.
In the VPC there was already a really simple Web Part Page that I wanted to test. I had not prepared the VPC myself and others had been using it before me (so I presumed that it would work) but no one had used it in a long while. When I attempted to login and access the Web Part page I received this exception:
Procedure or Function 'aspnet_PersonalizationPerUser_GetPageSettings' expects parameter '@TimeZoneAdjustment', which was not supplied.
I checked the aspnetdb database to see if there was something wrong there but the stored procedure did not contain a '@TimeZoneAdjustment' parameter... Some related forum discussions on the web indicated that it might be a problem with the solution and database being created under a pre-release version of .Net 2.0. I tried re-running aspnet_regsql to remove and recreate the aspnetdb database, I reinstalled .Net 2.0, I installed SP1 for .Net 2. Nothing helped.
It turns out that the default personalization provider for Web Parts uses a file based SQL Express database which is automatically created in the App_Data folder of the web application. This database had already been created using a previous version of the framework before I started looking at the solution. For the sake of confusion it is also named aspnetdb, removing this database from the App_Data folder solved the problem. When I logged in and opened the Web Part page the database was recreated with the correct structure and everything just worked!
Tuesday 09 January, 2007 (.Net | Team System)
Download the sample code.
I have tried different methods to move work items from one Team Foundation Server to another. The easiest method is doing it with Excel, but the problem is that I also want to keep all the work item history which Excel cannot do.
I found that the Revision property of each work item contains an array of complete copies of the work item as it appears at each revision. So iterating over each revision in order allows us to recreate each version that was saved! By setting the system time to the time of the original revision we can recreate a work item including all its changes, provided that we execute the code on our server so that the date changes are reflected when the work items are saved. An added bonus of iterating over each revision and saving them is that all the state changes are legal so we don't need to figure out how to get our work item to it's final state if we have complicated state transition rules.
If we execute our updates as a different user than the one that made the changes originally then each history entry will be tagged as "Edited by XXX on behalf of YYY". Unfortunately it appears that project reports are not updated with the historic data, all the new work items will be added to the reports the day that you import them.
The following code illustrated how I have copied almost all of the basic data from work items using revisions (unfortunately indentation was lost when I copy/pasted the code):
foreach (Revision r in wi.Revisions)
{
foreach(Field f in r.Fields)
{
if(f.ReferenceName == CoreFieldReferenceNames.ChangedDate)
{
System.Diagnostics.Debug.WriteLine(f.Name + " : "+ f.Value);
//Copy revision date
System.DateTime revDateTime = (System.DateTime)f.Value;
revSysTime.wDay = Convert.ToUInt16(revDateTime.Day);
revSysTime.wHour = Convert.ToUInt16(revDateTime.Hour);
revSysTime.wMilliseconds = Convert.ToUInt16(revDateTime.Millisecond)
revSysTime.wMinute = Convert.ToUInt16(revDateTime.Minute);
revSysTime.wMonth = Convert.ToUInt16(revDateTime.Month);
revSysTime.wSecond = Convert.ToUInt16(revDateTime.Second);
revSysTime.wYear = Convert.ToUInt16(revDateTime.Year);
}
if(f.ReferenceName != CoreFieldReferenceNames.History
&& f.ReferenceName != CoreFieldReferenceNames.WorkItemType
&& f.ReferenceName != CoreFieldReferenceNames.TeamProject
&& f.ReferenceName != CoreFieldReferenceNames.Id
&& f.ReferenceName != CoreFieldReferenceNames.IterationId
&& f.ReferenceName != CoreFieldReferenceNames.IterationPath
&& f.ReferenceName != CoreFieldReferenceNames.AreaId
&& f.ReferenceName != CoreFieldReferenceNames.AreaPath
&& f.ReferenceName != CoreFieldReferenceNames.CreatedBy
)
{
if(!f.IsComputed && f.IsValid)
{
destItem.Fields[f.Name].Value = f.Value;
}
}
}
System.Collections.ArrayList arrl = destItem.Validate();
if(arrl.Count > 0)
{
//Handle validation errors
}
else
{
GetSystemTime(ref currentSysTime);
SysTimeNeedsReset = true;
if(SetSystemTime(ref revSysTime) == 0)
{
throw new ApplicationException("SystemTime was not set!");
}
destItem.Save();
if(SetSystemTime(ref currentSysTime) == 0)
{
throw new ApplicationException("SystemTime was not reset!");
}
SysTimeNeedsReset = false;
}
}
You can download the code to the minimal WinForms app that I have created for copying work items here. The code is provided as-is with no guarantees or support...
Things that are not handled at all are: user mappings if the source and destination server have different users, area and iterations, links and attachments. There may be more things that are missing or don't work that haven't affected my setup.
Page 1 of 3 in the Net category Next Page
Disclaimer
The opinions expressed herein are my own personal opinions and do not represent
my employer's view in any way.
Sign In